Oct 31 12:59:08.490005 ip-10-0-141-83 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 12:59:08.873906 ip-10-0-141-83 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:59:08.873906 ip-10-0-141-83 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 12:59:08.873906 ip-10-0-141-83 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:59:08.873906 ip-10-0-141-83 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 12:59:08.877117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.876999    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:59:08.877117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877116    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:59:08.877117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877120    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877123    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877126    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877128    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877131    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877134    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877136    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877139    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877142    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877144    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877147    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877149    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877152    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877154    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877157    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877160    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877163    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877165    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877168    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:59:08.877211 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877170    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877173    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877175    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877178    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877180    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877186    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877190    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877192    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877195    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877197    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877200    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877202    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877205    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877208    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877211    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877214    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877217    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877219    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877222    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877225    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:59:08.877657 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877227    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877230    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877233    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877236    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877238    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877241    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877243    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877246    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877249    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877251    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877253    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877256    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877258    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877261    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877264    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877266    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877269    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877271    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877274    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877277    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:59:08.878152 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877279    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877281    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877284    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877286    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877290    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877293    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877295    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877298    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877302    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877306    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877309    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877312    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877314    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877317    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877319    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877324    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877327    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877330    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877332    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:59:08.878635 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877335    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877337    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877340    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877342    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877345    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877348    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877351    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877353    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877356    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877359    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877361    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877364    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877366    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877369    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877371    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877374    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877376    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877379    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877382    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877385    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:59:08.879095 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877387    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877391    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877393    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877396    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877399    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.877404    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878186    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878193    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878196    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878200    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878203    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878206    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878209    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878211    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878214    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878217    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878219    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878222    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:59:08.879563 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878224    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878227    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878229    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878232    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878235    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878237    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878240    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878242    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878245    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878247    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878250    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878253    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878255    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878260    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878262    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878265    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878267    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878270    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878273    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878275    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:59:08.880019 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878278    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878282    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878286    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878290    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878293    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878298    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878301    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878304    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878306    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878309    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878311    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878314    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878317    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878319    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878322    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878324    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878326    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878330    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878333    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:59:08.880491 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878335    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878338    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878340    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878343    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878346    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878348    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878351    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878354    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878356    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878359    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878361    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878364    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878366    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878369    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878371    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878373    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878376    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878378    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878382    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878384    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:59:08.880953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878387    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878389    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878391    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878394    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878396    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878398    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878402    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878406    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878408    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878411    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878413    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878416    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878418    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878421    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878423    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878426    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878428    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878431    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878434    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:59:08.881421 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878437    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878439    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878442    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878444    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878447    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878449    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878451    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878454    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878456    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878459    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878461    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878463    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878469    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878472    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878485    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.878488    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879048    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879055    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879058    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:59:08.881905 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879061    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879064    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879066    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879069    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879072    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879074    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879077    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879079    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879082    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879085    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879089    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879092    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879095    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879098    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879100    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879103    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879105    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879108    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879111    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:59:08.882492 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879113    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879116    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879118    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879121    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879123    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879126    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879129    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879132    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879134    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879137    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879139    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879142    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879144    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879146    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879149    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879152    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879154    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879157    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879159    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:59:08.882964 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879162    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879164    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879166    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879169    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879173    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879175    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879179    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879181    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879184    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879188    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879191    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879194    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879196    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879199    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879202    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879204    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879207    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879209    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879211    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:59:08.883437 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879214    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879217    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879219    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879222    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879224    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879227    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879230    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879232    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879235    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879237    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879239    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879242    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879245    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879247    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879250    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879252    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879255    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879259    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879263    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879267    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:59:08.883896 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879270    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879272    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879275    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879277    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879280    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879282    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879285    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879287    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879290    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879292    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879295    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879297    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879300    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879302    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879310    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879313    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879316    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879318    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879321    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879323    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:59:08.884356 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879326    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879328    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879331    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879333    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879337    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.879339    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880297    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880313    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880319    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880325    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880329    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880332    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880336    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880340    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880343    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880346    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880350    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880353    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880356    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880359    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880362    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880364    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880367    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880370    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 12:59:08.884842 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880378    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880381    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880383    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880392    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880396    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880400    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880402    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880405    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880409    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880412    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880414    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880417    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880420    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880423    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880427    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880430    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880433    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880436    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880439    2576 flags.go:64] FLAG: --enable-server="true"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880442    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880449    2576 flags.go:64] FLAG: --event-burst="100"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880452    2576 flags.go:64] FLAG: --event-qps="50"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880455    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880458    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 12:59:08.885407 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880460    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880465    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880468    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880471    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880473    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880476    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880479    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880482    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880485    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880488    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880491    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880494    2576 flags.go:64] FLAG: --feature-gates=""
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880506    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880509    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880512    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880515    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880518    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880521    2576 flags.go:64] FLAG: --help="false"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880524    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880527    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880530    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880533    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880536    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 12:59:08.886019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880540    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880543    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880545    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880548    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880551    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880554    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880557    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880559    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880562    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880565    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880568    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880571    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880574    2576 flags.go:64] FLAG: --lock-file=""
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880576    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880579    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880583    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880588    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880591    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880594    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880597    2576 flags.go:64] FLAG: --logging-format="text"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880599    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880603    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880611    2576 flags.go:64] FLAG: --manifest-url=""
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880614    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880618    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 12:59:08.886578 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880621    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880625    2576 flags.go:64] FLAG: --max-pods="110"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880628    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880631    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880633    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880636    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880639    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880642    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880645    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880653    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880655    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880658    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880661    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880664    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880667    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880670    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880672    2576 flags.go:64] FLAG: --port="10250"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880675    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880678    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0b4d28e464d8092f1"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880681    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880684    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880687    2576 flags.go:64] FLAG: --register-node="true"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880690    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880693    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880696    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880698    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 12:59:08.887361 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880713    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880723    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880726    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880729    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880737    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880740    2576 flags.go:64] FLAG: --runonce="false"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880743    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880746    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880749    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880751    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880754    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880757    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880762    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880765    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880769    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880772    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880775    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880778    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880781    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880784    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880786    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880792    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880795    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880797    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880804    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 31 12:59:08.888455 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880807    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880810    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880813    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880815    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880818    2576 flags.go:64] FLAG: --v="2"
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880822    2576 flags.go:64] FLAG: --version="false"
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880827    2576 flags.go:64] FLAG: --vmodule=""
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880831    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.880834    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881022    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881026    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881028    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881036    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881039    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881042    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881044    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881047    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881049    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881052    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881057    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881060    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:59:08.889559 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881063    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881066    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881068    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881071    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881073    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881076    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881078    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881081    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881083    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881086    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881088    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881091    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881093    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881096    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881099    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881101    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881104    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881107    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881109    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881112    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:59:08.890464 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881114    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881117    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881119    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881122    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881130    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881133    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881135    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881137    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881140    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881142    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881146    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881148    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881153    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881156    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881159    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881161    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881164    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881166    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881169    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:59:08.891351 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881171    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881175    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881178    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881181    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881184    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881186    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881189    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881191    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881194    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881196    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881199    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881201    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881204    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881207    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881209    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881211    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881214    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881216    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881224    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:59:08.892275 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881227    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881229    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881232    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881234    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881238    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881240    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881244    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881247    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881249    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881251    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881254    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881257    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881259    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881261    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881264    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881266    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881269    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881271    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881274    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881276    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:59:08.892998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881279    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881281    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881284    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881286    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881290    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881294    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881297    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881301    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881304    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881306    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881309    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881312    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881328    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881331    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881333    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881337    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:59:08.893869 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.881367    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881421    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881433    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881436    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881438    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881441    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881443    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881446    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881448    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881451    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881453    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881456    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881458    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881460    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881463    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881465    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881468    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881471    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881473    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:59:08.894552 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881476    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881478    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881481    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881483    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881486    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881488    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881491    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881493    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881496    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881498    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881506    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881509    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881511    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881514    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881517    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881521    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881523    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881526    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881529    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881531    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:59:08.895093 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881534    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881536    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881539    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881541    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881545    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881548    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881551    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881553    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881556    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881558    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881561    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881564    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881566    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881569    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881572    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881574    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881583    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881585    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881588    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:59:08.895847 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881590    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881593    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881596    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881598    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881607    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881609    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881612    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881616    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881620    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881623    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881625    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881628    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881630    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881633    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881635    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881637    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881640    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881642    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881645    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881647    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:59:08.896361 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881650    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881652    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881654    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881657    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881659    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881662    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881664    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881668    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881671    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881674    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881676    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881678    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881681    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881683    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881686    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881688    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881691    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881699    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881714    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:59:08.896884 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881717    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881721    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881724    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881726    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881729    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881731    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881734    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881736    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881739    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.881746    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.890297    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.890316    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.890339    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.890346    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890454    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890463    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890468    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890472    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890484    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:59:08.897390 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890489    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890494    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890498    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890502    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890506    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890510    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890514    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890519    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890526    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890532    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890536    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890550    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890556    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890560    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890564    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890569    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890573    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890578    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890582    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890607    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:59:08.897940 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890614    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890619    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890624    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890629    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890633    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890638    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890643    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890647    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890651    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890655    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890660    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890664    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890668    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890672    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890676    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890680    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890685    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890689    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890693    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:59:08.898792 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890697    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890718    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890723    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890727    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890732    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890744    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890749    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890753    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890757    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890762    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890766    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890771    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890775    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890779    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890783    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890789    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890795    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890801    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890805    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:59:08.899431 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890811    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890815    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890819    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890824    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890827    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890831    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890836    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890840    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890844    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890848    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890852    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890857    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890861    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890864    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890868    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890873    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890878    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890882    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890886    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890899    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:59:08.900117 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890911    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890915    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890920    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890924    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890928    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890932    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890936    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890940    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890944    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890948    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890953    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890957    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890961    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890965    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890969    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890973    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890977    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890982    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890986    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:59:08.900849 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890990    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890994    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.890998    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891002    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.891045    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891367    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891376    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891381    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891387    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891392    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891396    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891401    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891405    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891419    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891423    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891427    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:59:08.901485 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891432    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891436    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891440    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891444    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891448    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891452    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891457    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891461    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891465    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891469    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891473    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891477    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891481    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891486    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891490    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891494    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891498    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891503    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891507    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:59:08.902032 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891511    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891515    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891520    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891525    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891529    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891533    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891537    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891542    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891545    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891549    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891553    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891565    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891570    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891574    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891578    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891583    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891587    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891593    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891599    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:59:08.902699 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891605    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891610    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891614    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891619    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891623    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891627    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891631    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891635    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891639    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891643    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891647    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891651    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891656    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891660    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891664    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891668    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891672    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891677    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891681    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:59:08.903370 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891685    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891689    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891693    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891697    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891720    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891724    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891736    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891740    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891745    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891748    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891753    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891757    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891761    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891765    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891770    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891774    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891778    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891782    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891786    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891791    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:59:08.903953 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891794    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891798    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891803    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891808    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891813    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891817    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891821    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891825    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891829    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891833    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891837    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891841    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891846    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891852    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891857    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891861    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891865    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:59:08.904548 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891869    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.891909    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891992    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.891999    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892004    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892008    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892013    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892019    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892024    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892028    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892032    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892036    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892040    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892044    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892048    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 12:59:08.904987 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892053    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892056    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892061    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892065    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892069    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892074    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892078    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892082    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892086    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892090    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892094    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892098    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892102    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892106    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892110    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892115    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892119    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892122    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892126    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892130    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 12:59:08.905353 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892144    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892148    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892152    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892157    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892161    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892168    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892174    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892179    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892183    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892188    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892192    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892196    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892200    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892204    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892208    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892212    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892217    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892221    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892225    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 12:59:08.905826 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892229    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892233    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892237    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892242    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892246    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892250    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892254    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892258    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892262    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892267    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892271    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892275    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892279    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892283    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892294    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892298    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892303    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892307    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892311    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892315    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 12:59:08.906268 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892320    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892324    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892328    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892332    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892337    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892341    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892345    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892349    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892354    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892358    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892362    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892366    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892370    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892374    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892378    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892382    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892386    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892390    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892394    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 12:59:08.906816 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892401    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892407    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892421    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892426    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892430    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892434    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892438    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892442    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892454    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892459    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892463    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892467    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892471    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892475    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:08.892479    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.892558    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.895210    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.896051    2576 server.go:1014] "Starting client certificate rotation"
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.896188    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 12:59:08.907299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.896810    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 12:59:08.907768 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.898437    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:59:08.907768 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.904145    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 12:59:08.913452 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.913434    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 31 12:59:08.916751 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.916736    2576 log.go:25] "Validated CRI v1 image API"
Oct 31 12:59:08.919745 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.919697    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 12:59:08.922917 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.922888    2576 fs.go:135] Filesystem UUIDs: map[40fc6b03-0e55-4023-a6c4-c1a60c76cd84:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cffb27ce-fbee-4270-b589-ff6bfc791d69:/dev/nvme0n1p3]
Oct 31 12:59:08.923010 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.922916    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 12:59:08.928434 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.928302    2576 manager.go:217] Machine: {Timestamp:2026-10-31 12:59:08.926799489 +0000 UTC m=+0.323640863 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093577 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c97f6689dfe5d62b79f82166e09a SystemUUID:ec21c97f-6689-dfe5-d62b-79f82166e09a BootID:297b49ba-e85e-4562-9743-adbd2ab573d8 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:89:49:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:89:49:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:76:18:8c:cf:62:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 12:59:08.928925 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.928912    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 12:59:08.929084 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.929067    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 12:59:08.932193 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.932169    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 12:59:08.932366 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.932198    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-83.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 12:59:08.932410 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.932372    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 12:59:08.932410 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.932382    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 12:59:08.932410 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.932407    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:59:08.932483 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.932420    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 12:59:08.933265 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.933255    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 12:59:08.935168 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.935148    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 12:59:08.936831 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.936818    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 12:59:08.936968 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.936958    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 12:59:08.937004 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.936994    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 12:59:08.937038 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.937005    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 12:59:08.937038 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.937015    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 12:59:08.937038 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.937024    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 31 12:59:08.937038 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.937032    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 12:59:08.937389 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.937377    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:59:08.940666 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.939898    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 12:59:08.945956 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.945940    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 12:59:08.946825 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.946812    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 12:59:08.946879 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.946843    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 12:59:08.947012 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947001    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 12:59:08.947045 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947018    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 12:59:08.947045 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947024    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 12:59:08.947045 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947029    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 12:59:08.947045 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947037    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 12:59:08.947045 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947046    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 12:59:08.947557 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947548    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 12:59:08.947590 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947557    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 12:59:08.947590 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947570    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 12:59:08.947590 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947576    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 12:59:08.947590 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947584    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 12:59:08.947819 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.947593    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 12:59:08.948249 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.948232    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 12:59:08.948249 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.948243    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 12:59:08.951740 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.951723    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-83.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 12:59:08.952233 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.952220    2576 server.go:1263] "Started kubelet"
Oct 31 12:59:08.952342 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.952305    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 12:59:08.952453 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.952400    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 12:59:08.952540 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.952469    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 12:59:08.953199 ip-10-0-141-83 systemd[1]: Started Kubernetes Kubelet.
Oct 31 12:59:08.954456 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.954442    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 12:59:08.956295 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:08.956269    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 12:59:08.956393 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:08.956358    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-83.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 12:59:08.956794 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.956778    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 12:59:08.957609 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:08.957585    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 12:59:08.960286 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.960268    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2nzjd"
Oct 31 12:59:08.960695 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.960674    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 12:59:08.961652 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.961636    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 12:59:08.961762 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:08.961748    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:08.962228 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962210    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 12:59:08.962228 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962213    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 12:59:08.962353 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962237    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 12:59:08.962353 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962306    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 12:59:08.962353 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962316    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 12:59:08.962448 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:08.961399    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-83.us-east-2.compute.internal.18e39eae812b58ed  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-83.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:59:08.952189165 +0000 UTC m=+0.349030541,LastTimestamp:2026-10-31 12:59:08.952189165 +0000 UTC m=+0.349030541,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-83.us-east-2.compute.internal,}"
Oct 31 12:59:08.962506 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962468    2576 factory.go:55] Registering systemd factory
Oct 31 12:59:08.962506 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962490    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 31 12:59:08.962686 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962674    2576 factory.go:153] Registering CRI-O factory
Oct 31 12:59:08.962755 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962689    2576 factory.go:223] Registration of the crio container factory successfully
Oct 31 12:59:08.962787 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962756    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 12:59:08.962787 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962775    2576 factory.go:103] Registering Raw factory
Oct 31 12:59:08.962841 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.962791    2576 manager.go:1199] Started watching for new ooms in manager
Oct 31 12:59:08.964075 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.964058    2576 manager.go:319] Starting recovery of all containers
Oct 31 12:59:08.966219 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:08.966191    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-83.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 12:59:08.971260 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:08.971225    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 12:59:08.972968 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.972944    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2nzjd"
Oct 31 12:59:08.976104 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.976008    2576 manager.go:324] Recovery completed
Oct 31 12:59:08.981374 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.981360    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:59:08.983682 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.983664    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:59:08.983796 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.983721    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:59:08.983796 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.983734    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:59:08.984285 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.984270    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 31 12:59:08.984285 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.984282    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 12:59:08.984392 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.984302    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 12:59:08.984392 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.984313    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:59:08.985081 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.985067    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:59:08.985140 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.985096    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:59:08.985140 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.985111    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:59:08.986270 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:08.986183    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-83.us-east-2.compute.internal.18e39eae830bdf20  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-83.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-83.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-83.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 12:59:08.9836808 +0000 UTC m=+0.380522173,LastTimestamp:2026-10-31 12:59:08.9836808 +0000 UTC m=+0.380522173,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-83.us-east-2.compute.internal,}"
Oct 31 12:59:08.986415 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.986401    2576 policy_none.go:50] "Start"
Oct 31 12:59:08.986471 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.986420    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 12:59:08.986471 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.986433    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 12:59:08.988397 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:08.988384    2576 policy_none.go:44] "Start"
Oct 31 12:59:09.025430 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.025238    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 12:59:09.029331 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.029316    2576 manager.go:341] "Starting Device Plugin manager"
Oct 31 12:59:09.029928 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.029913    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 12:59:09.029983 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.029932    2576 server.go:87] "Starting device plugin registration server"
Oct 31 12:59:09.030186 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.030173    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 12:59:09.030261 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.030187    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 12:59:09.030371 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.030300    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 12:59:09.030497 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.030401    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 12:59:09.030497 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.030410    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 12:59:09.030943 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.030924    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 12:59:09.031013 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.030967    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.087268 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.087223    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 12:59:09.089388 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.088605    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 12:59:09.089388 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.088625    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 12:59:09.089388 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.088647    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 12:59:09.089388 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.088679    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 12:59:09.093415 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.093396    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:59:09.130614 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.130550    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:59:09.131566 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.131550    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:59:09.131619 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.131587    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:59:09.131619 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.131600    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:59:09.131679 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.131626    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.141782 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.141765    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.141837 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.141791    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-83.us-east-2.compute.internal\": node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.159558 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.159533    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.189367 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.189338    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"]
Oct 31 12:59:09.189501 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.189399    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:59:09.193337 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.193322    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:59:09.193397 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.193347    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:59:09.193397 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.193358    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:59:09.196938 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.196925    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:59:09.197159 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.197138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.197275 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.197178    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:59:09.197716 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.197686    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:59:09.197795 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.197735    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:59:09.197795 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.197750    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:59:09.200646 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.200628    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:59:09.200741 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.200662    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:59:09.200741 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.200672    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:59:09.203996 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.203984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.204050 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.204009    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 12:59:09.204754 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.204739    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 12:59:09.204819 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.204773    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 12:59:09.204819 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.204787    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 12:59:09.217338 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.217307    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found" node="ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.221989 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.221973    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found" node="ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.260422 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.260393    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.264741 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.264725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.264833 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.264749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.264833 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.264767    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ebdbb4e6bd986d2f09f581a50c7d1f8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"7ebdbb4e6bd986d2f09f581a50c7d1f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.361183 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.361143    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.365539 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.365509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.365682 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.365547    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.365682 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.365565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ebdbb4e6bd986d2f09f581a50c7d1f8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"7ebdbb4e6bd986d2f09f581a50c7d1f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.365682 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.365622    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.365682 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.365631    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.365682 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.365669    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ebdbb4e6bd986d2f09f581a50c7d1f8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"7ebdbb4e6bd986d2f09f581a50c7d1f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.462024 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.461929    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.520372 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.520343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.525310 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.525145    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:09.562531 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.562501    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.662962 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.662929    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.763421 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.763357    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.863820 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.863792    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.896120 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.896095    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 12:59:09.896582 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.896243    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 12:59:09.964522 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:09.964491    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:09.978852 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.978806    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:54:08 +0000 UTC" deadline="2028-08-03 21:57:26.828635954 +0000 UTC"
Oct 31 12:59:09.978852 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:09.978847    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15416h58m16.849791498s"
Oct 31 12:59:10.064967 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:10.064936    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:10.072923 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.072903    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:59:10.134146 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:10.134108    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebdbb4e6bd986d2f09f581a50c7d1f8.slice/crio-9dfe419f153032b25936bd01505811562dc0b0d58b0be0e406f90a9fee7bb889 WatchSource:0}: Error finding container 9dfe419f153032b25936bd01505811562dc0b0d58b0be0e406f90a9fee7bb889: Status 404 returned error can't find the container with id 9dfe419f153032b25936bd01505811562dc0b0d58b0be0e406f90a9fee7bb889
Oct 31 12:59:10.138514 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.138498    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 12:59:10.165272 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:10.165244    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:10.259310 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:10.259282    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3754ca7849e18e0d20b7fa7b829435b5.slice/crio-460569fc828b21f2b6acdb11f345c732ec3f9572f34eaa5f26c3d38e05778327 WatchSource:0}: Error finding container 460569fc828b21f2b6acdb11f345c732ec3f9572f34eaa5f26c3d38e05778327: Status 404 returned error can't find the container with id 460569fc828b21f2b6acdb11f345c732ec3f9572f34eaa5f26c3d38e05778327
Oct 31 12:59:10.265356 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:10.265339    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:10.366013 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:10.365917    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:10.392771 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.392747    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:59:10.466318 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:10.466278    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 12:59:10.535520 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.535492    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:59:10.562605 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.562569    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:10.575676 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.575655    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:59:10.576653 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.576641    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 12:59:10.586550 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.586533    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 12:59:10.870679 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.870644    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 12:59:10.937975 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.937946    2576 apiserver.go:52] "Watching apiserver"
Oct 31 12:59:10.945119 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.945096    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 12:59:10.945405 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.945390    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-7pzvw","openshift-network-operator/iptables-alerter-4p8sf","kube-system/konnectivity-agent-phbg5","kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-6rkkf","openshift-multus/network-metrics-daemon-jbb9m","openshift-network-diagnostics/network-check-target-r4bj6","kube-system/global-pull-secret-syncer-c6hpz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb","openshift-ovn-kubernetes/ovnkube-node-cbknb","openshift-cluster-node-tuning-operator/tuned-5pwr7","openshift-image-registry/node-ca-lvtzl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal","openshift-dns/node-resolver-ld8vz"]
Oct 31 12:59:10.948049 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.948023    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.950253 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.950237    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.952165 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.952144    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:10.952585 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.952546    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 12:59:10.952909 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.952822    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hwtqn\""
Oct 31 12:59:10.952909 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.952861    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 12:59:10.952909 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.952867    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 12:59:10.952909 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.952865    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 12:59:10.952909 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.952904    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 12:59:10.953128 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.952968    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 12:59:10.953645 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.953628    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:59:10.953645 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.953632    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 12:59:10.953826 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.953694    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mw6wr\""
Oct 31 12:59:10.954320 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.954302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:10.955777 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.955759    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 12:59:10.955861 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.955849    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 12:59:10.955961 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.955948    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5b9kz\""
Oct 31 12:59:10.957042 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.957007    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:10.958055 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.957732    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 12:59:10.958055 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.957954    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 12:59:10.958310 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.958074    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2xlcn\""
Oct 31 12:59:10.959542 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.959518    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:10.959621 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:10.959598    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:10.961375 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.961353    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 12:59:10.961490 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.961435    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 12:59:10.961696 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.961679    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 12:59:10.961809 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.961743    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 12:59:10.961809 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.961784    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 12:59:10.961913 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.961844    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 12:59:10.961913 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.961903    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2kvxk\""
Oct 31 12:59:10.962003 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.961975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:10.964098 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.964077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:10.964185 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:10.964154    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:10.966399 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.966381    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:10.970381 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.970364    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:10.971849 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.971831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-slash\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.971942 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.971862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-config\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.971942 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.971888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-env-overrides\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.971942 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.971910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztx68\" (UniqueName: \"kubernetes.io/projected/87328843-fa88-4d5d-9904-5a3036d5bb10-kube-api-access-ztx68\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.971942 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.971934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-system-cni-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:10.972133 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:10.972133 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-host\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.972133 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:10.972133 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-device-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:10.972335 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-log-socket\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.972335 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:10.972335 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:10.972335 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.972335 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.972335 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972278    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-dbus\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:10.972335 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-tuned\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-systemd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-cnibin\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-run\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-lib-modules\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-tmp\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/448c900f-a9c5-4df1-b49e-e76269934da4-hosts-file\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-netns\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972627    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-netd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972733    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:10.972788 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972767    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-kubelet-config\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972814    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-systemd\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-node-log\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972908    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-kubernetes\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zkqv\" (UniqueName: \"kubernetes.io/projected/1e7e4656-607a-4995-a250-a3bda30be0c1-kube-api-access-9zkqv\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:10.972956    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.972962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/686b38d0-13b9-4d2c-a1cf-5b8053804350-konnectivity-ca\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973000    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448c900f-a9c5-4df1-b49e-e76269934da4-tmp-dir\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-var-lib-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973115    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-etc-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87328843-fa88-4d5d-9904-5a3036d5bb10-ovn-node-metrics-cert\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973155    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-modprobe-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/686b38d0-13b9-4d2c-a1cf-5b8053804350-agent-certs\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55jgb\" (UniqueName: \"kubernetes.io/projected/448c900f-a9c5-4df1-b49e-e76269934da4-kube-api-access-55jgb\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:10.973299 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8pj\" (UniqueName: \"kubernetes.io/projected/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kube-api-access-4h8pj\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-ovn\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-var-lib-kubelet\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6lbt\" (UniqueName: \"kubernetes.io/projected/e3f637a5-1cd2-41a6-98d6-ad45904012ad-kube-api-access-k6lbt\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysconfig\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973464    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-socket-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-registration-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973522    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-kubelet\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgzj5\" (UniqueName: \"kubernetes.io/projected/5510e6da-dd62-4c2f-b73e-75ca74cad962-kube-api-access-lgzj5\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-conf\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-sys-fs\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-bin\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973834    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-script-lib\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-sys\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973916    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:10.974121 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.973946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-systemd-units\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:10.975177 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:10.975158    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.000773 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.000749    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 12:59:11.000880 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.000821    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-854sj"
Oct 31 12:59:11.015619 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.015587    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-854sj"
Oct 31 12:59:11.049795 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.049763    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 12:59:11.049795 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.049769    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 12:59:11.049795 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.049777    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 12:59:11.050076 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.049857    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 12:59:11.050076 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.049920    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 12:59:11.050076 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.049948    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 12:59:11.050226 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.050082    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 12:59:11.050226 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.050085    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 12:59:11.050226 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.050154    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 12:59:11.050226 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.050202    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7lhn5\""
Oct 31 12:59:11.050226 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.050208    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2x4wq\""
Oct 31 12:59:11.050399 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.050226    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vtxsh\""
Oct 31 12:59:11.050399 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.050237    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 12:59:11.050399 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.050264    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drb8j\""
Oct 31 12:59:11.063126 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.063102    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 12:59:11.074152 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074080    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.074286 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c4a59dc-f603-4d34-a82a-23813a1b86b6-iptables-alerter-script\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.074286 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074195    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-host\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.074286 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-k8s-cni-cncf-io\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.074286 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-netns\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.074286 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.074523 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-device-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.074523 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-host\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.074523 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-kubelet\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.074523 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wj9\" (UniqueName: \"kubernetes.io/projected/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-kube-api-access-w4wj9\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.074523 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-log-socket\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.074523 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074400    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-log-socket\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.074523 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074512    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-device-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.074830 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:11.074830 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074610    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.074830 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.074725    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:11.074830 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074753    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.074830 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c4a59dc-f603-4d34-a82a-23813a1b86b6-host-slash\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.074830 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.074807    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs podName:e3f637a5-1cd2-41a6-98d6-ad45904012ad nodeName:}" failed. No retries permitted until 2026-10-31 12:59:11.574790724 +0000 UTC m=+2.971632103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs") pod "network-metrics-daemon-jbb9m" (UID: "e3f637a5-1cd2-41a6-98d6-ad45904012ad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:11.075117 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ts4m\" (UniqueName: \"kubernetes.io/projected/5c4a59dc-f603-4d34-a82a-23813a1b86b6-kube-api-access-2ts4m\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.075117 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.075117 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.075117 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-dbus\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:11.075117 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.074994    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.075117 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075016    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.075117 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-tuned\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075131    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075181    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-dbus\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075192    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-socket-dir-parent\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-systemd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-cnibin\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-run\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075313    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-cnibin\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075319    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-run\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075324    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-lib-modules\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.075363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075356    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-systemd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-tmp\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/448c900f-a9c5-4df1-b49e-e76269934da4-hosts-file\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075448    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075468    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-lib-modules\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cnibin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075515    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/448c900f-a9c5-4df1-b49e-e76269934da4-hosts-file\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075519    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gztt\" (UniqueName: \"kubernetes.io/projected/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-kube-api-access-4gztt\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-netns\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-netd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-netns\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-kubelet-config\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-systemd\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-system-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cni-binary-copy\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.075821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-node-log\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075835    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-netd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-kubernetes\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkqv\" (UniqueName: \"kubernetes.io/projected/1e7e4656-607a-4995-a250-a3bda30be0c1-kube-api-access-9zkqv\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/686b38d0-13b9-4d2c-a1cf-5b8053804350-konnectivity-ca\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448c900f-a9c5-4df1-b49e-e76269934da4-tmp-dir\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-bin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-host\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.075992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-var-lib-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076063    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-multus\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-serviceca\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076094    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-kubernetes\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076105    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076118    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-etc-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076144    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-kubelet-config\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:11.076565 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87328843-fa88-4d5d-9904-5a3036d5bb10-ovn-node-metrics-cert\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-modprobe-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076185    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-systemd\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/686b38d0-13b9-4d2c-a1cf-5b8053804350-agent-certs\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jgb\" (UniqueName: \"kubernetes.io/projected/448c900f-a9c5-4df1-b49e-e76269934da4-kube-api-access-55jgb\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076269    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-node-log\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076276    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8pj\" (UniqueName: \"kubernetes.io/projected/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kube-api-access-4h8pj\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-ovn\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076324    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-var-lib-kubelet\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lbt\" (UniqueName: \"kubernetes.io/projected/e3f637a5-1cd2-41a6-98d6-ad45904012ad-kube-api-access-k6lbt\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-conf-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-daemon-config\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysconfig\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-socket-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-registration-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076490    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-hostroot\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-etc-kubernetes\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.077247 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-kubelet\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076557    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzj5\" (UniqueName: \"kubernetes.io/projected/5510e6da-dd62-4c2f-b73e-75ca74cad962-kube-api-access-lgzj5\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-conf\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-sys-fs\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-multus-certs\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-bin\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076762    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-script-lib\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076768    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/686b38d0-13b9-4d2c-a1cf-5b8053804350-konnectivity-ca\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-sys\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-systemd-units\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076863    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-slash\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-config\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-env-overrides\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076909    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztx68\" (UniqueName: \"kubernetes.io/projected/87328843-fa88-4d5d-9904-5a3036d5bb10-kube-api-access-ztx68\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078108 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-system-cni-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.076978    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-system-cni-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.077211    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448c900f-a9c5-4df1-b49e-e76269934da4-tmp-dir\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.077296    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-modprobe-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.077316    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-var-lib-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.077355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.077422    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.077473    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret podName:97d66a46-02bb-4220-a78c-850e615072ca nodeName:}" failed. No retries permitted until 2026-10-31 12:59:11.577459677 +0000 UTC m=+2.974301038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret") pod "global-pull-secret-syncer-c6hpz" (UID: "97d66a46-02bb-4220-a78c-850e615072ca") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.077542    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-etc-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.077831    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-slash\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.077892    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-sys\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.077935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.077969    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-systemd-units\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078060    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-script-lib\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078298    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-env-overrides\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078363    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-var-lib-kubelet\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078301    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-registration-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.078866 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078403    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-conf\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.079602 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078423    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-ovn\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.079602 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078465    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-kubelet\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.079602 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078544    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysconfig\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.079602 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078734    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-socket-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.079602 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078764    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-bin\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.079602 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.078814    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-sys-fs\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.079602 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.079042    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-config\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.079602 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.079122    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.080482 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.080354    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-tuned\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.080482 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.080465    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87328843-fa88-4d5d-9904-5a3036d5bb10-ovn-node-metrics-cert\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.080646 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.080616    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-tmp\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.081003 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.080960    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/686b38d0-13b9-4d2c-a1cf-5b8053804350-agent-certs\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:11.090860 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.090825    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztx68\" (UniqueName: \"kubernetes.io/projected/87328843-fa88-4d5d-9904-5a3036d5bb10-kube-api-access-ztx68\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.094020 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.094001    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkqv\" (UniqueName: \"kubernetes.io/projected/1e7e4656-607a-4995-a250-a3bda30be0c1-kube-api-access-9zkqv\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.094430 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.094384    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"3754ca7849e18e0d20b7fa7b829435b5","Type":"ContainerStarted","Data":"460569fc828b21f2b6acdb11f345c732ec3f9572f34eaa5f26c3d38e05778327"}
Oct 31 12:59:11.095751 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.095730    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"7ebdbb4e6bd986d2f09f581a50c7d1f8","Type":"ContainerStarted","Data":"9dfe419f153032b25936bd01505811562dc0b0d58b0be0e406f90a9fee7bb889"}
Oct 31 12:59:11.098574 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.098403    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8pj\" (UniqueName: \"kubernetes.io/projected/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kube-api-access-4h8pj\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.098574 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.098531    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jgb\" (UniqueName: \"kubernetes.io/projected/448c900f-a9c5-4df1-b49e-e76269934da4-kube-api-access-55jgb\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:11.098858 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.098836    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzj5\" (UniqueName: \"kubernetes.io/projected/5510e6da-dd62-4c2f-b73e-75ca74cad962-kube-api-access-lgzj5\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.100256 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.100233    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6lbt\" (UniqueName: \"kubernetes.io/projected/e3f637a5-1cd2-41a6-98d6-ad45904012ad-kube-api-access-k6lbt\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c4a59dc-f603-4d34-a82a-23813a1b86b6-iptables-alerter-script\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-k8s-cni-cncf-io\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-netns\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-kubelet\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wj9\" (UniqueName: \"kubernetes.io/projected/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-kube-api-access-w4wj9\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c4a59dc-f603-4d34-a82a-23813a1b86b6-host-slash\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ts4m\" (UniqueName: \"kubernetes.io/projected/5c4a59dc-f603-4d34-a82a-23813a1b86b6-kube-api-access-2ts4m\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177575    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-netns\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-socket-dir-parent\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cnibin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177643    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gztt\" (UniqueName: \"kubernetes.io/projected/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-kube-api-access-4gztt\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177668    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-system-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177696    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cni-binary-copy\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177760    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-bin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-host\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-multus\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.178743 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177837    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-serviceca\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-conf-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-daemon-config\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-hostroot\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-etc-kubernetes\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.177974    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-multus-certs\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178032    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c4a59dc-f603-4d34-a82a-23813a1b86b6-iptables-alerter-script\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-k8s-cni-cncf-io\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178394    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c4a59dc-f603-4d34-a82a-23813a1b86b6-host-slash\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178412    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-kubelet\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178420    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-conf-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178450    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-bin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178499    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-hostroot\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178548    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-socket-dir-parent\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178560    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-system-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178574    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cnibin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.179760 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178590    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-host\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.180651 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178599    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-etc-kubernetes\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.180651 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178606    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-multus\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.180651 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178635    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-multus-certs\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.180651 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.178784    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-serviceca\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.180651 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.179632    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cni-binary-copy\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.180651 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.179641    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-daemon-config\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.188638 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.188613    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:59:11.188638 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.188638    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:59:11.188857 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.188651    2576 projected.go:196] Error preparing data for projected volume kube-api-access-t6db7 for pod openshift-network-diagnostics/network-check-target-r4bj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:11.188857 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.188739    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7 podName:3c98a803-2cc2-4806-8a14-b05c31ee7f03 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:11.688721281 +0000 UTC m=+3.085562646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t6db7" (UniqueName: "kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7") pod "network-check-target-r4bj6" (UID: "3c98a803-2cc2-4806-8a14-b05c31ee7f03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:11.191689 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.191656    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ts4m\" (UniqueName: \"kubernetes.io/projected/5c4a59dc-f603-4d34-a82a-23813a1b86b6-kube-api-access-2ts4m\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.191689 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.191671    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wj9\" (UniqueName: \"kubernetes.io/projected/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-kube-api-access-w4wj9\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.191963 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.191945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gztt\" (UniqueName: \"kubernetes.io/projected/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-kube-api-access-4gztt\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.278363 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.278328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 12:59:11.280817 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.280795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:11.286005 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.285972    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:11.287019 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.286998    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ld8vz"
Oct 31 12:59:11.295245 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:11.295219    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e7e4656_607a_4995_a250_a3bda30be0c1.slice/crio-87c9a1a38fd3d9ec8ebc970e7953e04f66bbb88cbad19e43b9ca2471ef31c12a WatchSource:0}: Error finding container 87c9a1a38fd3d9ec8ebc970e7953e04f66bbb88cbad19e43b9ca2471ef31c12a: Status 404 returned error can't find the container with id 87c9a1a38fd3d9ec8ebc970e7953e04f66bbb88cbad19e43b9ca2471ef31c12a
Oct 31 12:59:11.298851 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:11.298823    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686b38d0_13b9_4d2c_a1cf_5b8053804350.slice/crio-24b898120cb835f08c3532e4b6190751f2455f3a41a6d3d9c1d4c43561ba438f WatchSource:0}: Error finding container 24b898120cb835f08c3532e4b6190751f2455f3a41a6d3d9c1d4c43561ba438f: Status 404 returned error can't find the container with id 24b898120cb835f08c3532e4b6190751f2455f3a41a6d3d9c1d4c43561ba438f
Oct 31 12:59:11.301451 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:11.301421    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87328843_fa88_4d5d_9904_5a3036d5bb10.slice/crio-b5d2d2ff275207a201d341e17482e44f97e6fe0cd09acce772a66e03331a8a79 WatchSource:0}: Error finding container b5d2d2ff275207a201d341e17482e44f97e6fe0cd09acce772a66e03331a8a79: Status 404 returned error can't find the container with id b5d2d2ff275207a201d341e17482e44f97e6fe0cd09acce772a66e03331a8a79
Oct 31 12:59:11.303378 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:11.303356    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448c900f_a9c5_4df1_b49e_e76269934da4.slice/crio-ee22a4e32efe46d0530853935902db4730f0f67b9f487ffefa275ec02aa40665 WatchSource:0}: Error finding container ee22a4e32efe46d0530853935902db4730f0f67b9f487ffefa275ec02aa40665: Status 404 returned error can't find the container with id ee22a4e32efe46d0530853935902db4730f0f67b9f487ffefa275ec02aa40665
Oct 31 12:59:11.305081 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.305061    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 12:59:11.314812 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.314599    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 12:59:11.315521 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:11.315499    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5510e6da_dd62_4c2f_b73e_75ca74cad962.slice/crio-e1927855abaceab46e328d6f0151d935a379dbc4633a1cfb9704de5d825d242d WatchSource:0}: Error finding container e1927855abaceab46e328d6f0151d935a379dbc4633a1cfb9704de5d825d242d: Status 404 returned error can't find the container with id e1927855abaceab46e328d6f0151d935a379dbc4633a1cfb9704de5d825d242d
Oct 31 12:59:11.316543 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.316521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 12:59:11.318874 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.318856    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7pzvw"
Oct 31 12:59:11.323341 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.323312    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 12:59:11.328125 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:11.328090    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c4a59dc_f603_4d34_a82a_23813a1b86b6.slice/crio-e403741246066bdfc00ee3aba1ca9f28c33660c88723714e5a8c4b68be50de39 WatchSource:0}: Error finding container e403741246066bdfc00ee3aba1ca9f28c33660c88723714e5a8c4b68be50de39: Status 404 returned error can't find the container with id e403741246066bdfc00ee3aba1ca9f28c33660c88723714e5a8c4b68be50de39
Oct 31 12:59:11.331321 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:11.331293    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5337ff71_42fc_4f7d_a92f_67e13f9c0a00.slice/crio-804915a6fb388953ad84b69aa7fae1db20f2c525906b4282360e1e5be9df008d WatchSource:0}: Error finding container 804915a6fb388953ad84b69aa7fae1db20f2c525906b4282360e1e5be9df008d: Status 404 returned error can't find the container with id 804915a6fb388953ad84b69aa7fae1db20f2c525906b4282360e1e5be9df008d
Oct 31 12:59:11.335239 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:11.335216    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22c3df7_5789_4439_ba22_3a7d2b30c3fa.slice/crio-87b7996c7cfc5cc6f7d60ae9b4cc8f0d0e44a0020897de5a8ee0585bc86a20c5 WatchSource:0}: Error finding container 87b7996c7cfc5cc6f7d60ae9b4cc8f0d0e44a0020897de5a8ee0585bc86a20c5: Status 404 returned error can't find the container with id 87b7996c7cfc5cc6f7d60ae9b4cc8f0d0e44a0020897de5a8ee0585bc86a20c5
Oct 31 12:59:11.580647 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.580559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:11.580825 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.580652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:11.580825 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.580742    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:11.580825 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.580799    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:11.580825 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.580824    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs podName:e3f637a5-1cd2-41a6-98d6-ad45904012ad nodeName:}" failed. No retries permitted until 2026-10-31 12:59:12.580802213 +0000 UTC m=+3.977643575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs") pod "network-metrics-daemon-jbb9m" (UID: "e3f637a5-1cd2-41a6-98d6-ad45904012ad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:11.580965 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.580852    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret podName:97d66a46-02bb-4220-a78c-850e615072ca nodeName:}" failed. No retries permitted until 2026-10-31 12:59:12.58083711 +0000 UTC m=+3.977678489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret") pod "global-pull-secret-syncer-c6hpz" (UID: "97d66a46-02bb-4220-a78c-850e615072ca") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:11.782151 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:11.782112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:11.782302 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.782259    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:59:11.782302 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.782276    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:59:11.782302 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.782289    2576 projected.go:196] Error preparing data for projected volume kube-api-access-t6db7 for pod openshift-network-diagnostics/network-check-target-r4bj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:11.782398 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:11.782348    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7 podName:3c98a803-2cc2-4806-8a14-b05c31ee7f03 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:12.782330373 +0000 UTC m=+4.179171779 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6db7" (UniqueName: "kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7") pod "network-check-target-r4bj6" (UID: "3c98a803-2cc2-4806-8a14-b05c31ee7f03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:12.020984 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.020245    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:54:10 +0000 UTC" deadline="2028-05-27 22:09:06.57328246 +0000 UTC"
Oct 31 12:59:12.020984 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.020307    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13785h9m54.552980018s"
Oct 31 12:59:12.090374 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.089643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:12.090374 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.089797    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:12.090374 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.090203    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:12.090374 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.090307    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:12.124068 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.123941    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7pzvw" event={"ID":"5337ff71-42fc-4f7d-a92f-67e13f9c0a00","Type":"ContainerStarted","Data":"804915a6fb388953ad84b69aa7fae1db20f2c525906b4282360e1e5be9df008d"}
Oct 31 12:59:12.135198 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.135095    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4p8sf" event={"ID":"5c4a59dc-f603-4d34-a82a-23813a1b86b6","Type":"ContainerStarted","Data":"e403741246066bdfc00ee3aba1ca9f28c33660c88723714e5a8c4b68be50de39"}
Oct 31 12:59:12.145559 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.145519    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" event={"ID":"5510e6da-dd62-4c2f-b73e-75ca74cad962","Type":"ContainerStarted","Data":"e1927855abaceab46e328d6f0151d935a379dbc4633a1cfb9704de5d825d242d"}
Oct 31 12:59:12.159509 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.159450    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ld8vz" event={"ID":"448c900f-a9c5-4df1-b49e-e76269934da4","Type":"ContainerStarted","Data":"ee22a4e32efe46d0530853935902db4730f0f67b9f487ffefa275ec02aa40665"}
Oct 31 12:59:12.168431 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.168397    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-phbg5" event={"ID":"686b38d0-13b9-4d2c-a1cf-5b8053804350","Type":"ContainerStarted","Data":"24b898120cb835f08c3532e4b6190751f2455f3a41a6d3d9c1d4c43561ba438f"}
Oct 31 12:59:12.177977 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.177944    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb" event={"ID":"5b16b0cb-c872-4fee-8a2a-83adb2777cc6","Type":"ContainerStarted","Data":"b1f5c29d61c9b98382520999638ed0794183b0998ffe6cdd3d6b9560947539b9"}
Oct 31 12:59:12.181962 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.181929    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" event={"ID":"87328843-fa88-4d5d-9904-5a3036d5bb10","Type":"ContainerStarted","Data":"b5d2d2ff275207a201d341e17482e44f97e6fe0cd09acce772a66e03331a8a79"}
Oct 31 12:59:12.190783 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.190754    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5pwr7" event={"ID":"1e7e4656-607a-4995-a250-a3bda30be0c1","Type":"ContainerStarted","Data":"87c9a1a38fd3d9ec8ebc970e7953e04f66bbb88cbad19e43b9ca2471ef31c12a"}
Oct 31 12:59:12.205881 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.205843    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"7ebdbb4e6bd986d2f09f581a50c7d1f8","Type":"ContainerStarted","Data":"c9224fb1fce494c15fd6516e0acf6417d336adc8131437e8b08cb2a4cb8862d5"}
Oct 31 12:59:12.210013 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.209783    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lvtzl" event={"ID":"e22c3df7-5789-4439-ba22-3a7d2b30c3fa","Type":"ContainerStarted","Data":"87b7996c7cfc5cc6f7d60ae9b4cc8f0d0e44a0020897de5a8ee0585bc86a20c5"}
Oct 31 12:59:12.228316 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.228049    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal" podStartSLOduration=2.228033007 podStartE2EDuration="2.228033007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:12.227426748 +0000 UTC m=+3.624268132" watchObservedRunningTime="2026-10-31 12:59:12.228033007 +0000 UTC m=+3.624874392"
Oct 31 12:59:12.589613 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.589275    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:12.589613 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.589606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:12.589844 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.589749    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:12.589844 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.589814    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs podName:e3f637a5-1cd2-41a6-98d6-ad45904012ad nodeName:}" failed. No retries permitted until 2026-10-31 12:59:14.589795859 +0000 UTC m=+5.986637238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs") pod "network-metrics-daemon-jbb9m" (UID: "e3f637a5-1cd2-41a6-98d6-ad45904012ad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:12.592969 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.590187    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:12.592969 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.590231    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret podName:97d66a46-02bb-4220-a78c-850e615072ca nodeName:}" failed. No retries permitted until 2026-10-31 12:59:14.590218258 +0000 UTC m=+5.987059634 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret") pod "global-pull-secret-syncer-c6hpz" (UID: "97d66a46-02bb-4220-a78c-850e615072ca") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:12.792092 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:12.792050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:12.792264 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.792238    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:59:12.792264 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.792258    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:59:12.792365 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.792269    2576 projected.go:196] Error preparing data for projected volume kube-api-access-t6db7 for pod openshift-network-diagnostics/network-check-target-r4bj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:12.792365 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:12.792328    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7 podName:3c98a803-2cc2-4806-8a14-b05c31ee7f03 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:14.792310604 +0000 UTC m=+6.189151980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6db7" (UniqueName: "kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7") pod "network-check-target-r4bj6" (UID: "3c98a803-2cc2-4806-8a14-b05c31ee7f03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:13.021165 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:13.021067    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:54:10 +0000 UTC" deadline="2028-04-07 11:22:30.186870005 +0000 UTC"
Oct 31 12:59:13.021165 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:13.021130    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12574h23m17.165744431s"
Oct 31 12:59:13.090766 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:13.090281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:13.090766 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:13.090404    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:13.235328 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:13.235199    2576 generic.go:356] "Generic (PLEG): container finished" podID="3754ca7849e18e0d20b7fa7b829435b5" containerID="37bb83e7caa761b220e06ff2abea8f856dd681baff02af97deb6f50cc95735af" exitCode=0
Oct 31 12:59:13.235328 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:13.235251    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"3754ca7849e18e0d20b7fa7b829435b5","Type":"ContainerDied","Data":"37bb83e7caa761b220e06ff2abea8f856dd681baff02af97deb6f50cc95735af"}
Oct 31 12:59:14.089671 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:14.088970    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:14.089671 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.089122    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:14.089671 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:14.089493    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:14.089671 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.089589    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:14.248755 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:14.248413    2576 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="45583a5d6f0b23709f4d7d26e53fb46a2ba3e0b8cbdc83a50b2c4ca34013ba5e" exitCode=0
Oct 31 12:59:14.248755 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:14.248667    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" event={"ID":"5510e6da-dd62-4c2f-b73e-75ca74cad962","Type":"ContainerDied","Data":"45583a5d6f0b23709f4d7d26e53fb46a2ba3e0b8cbdc83a50b2c4ca34013ba5e"}
Oct 31 12:59:14.257502 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:14.257478    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"3754ca7849e18e0d20b7fa7b829435b5","Type":"ContainerStarted","Data":"241f8fe09639f15e97080acac5cd0be0a43235bb4de569439d579b81482c043e"}
Oct 31 12:59:14.609608 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:14.609577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:14.609821 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:14.609632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:14.609821 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.609747    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:14.609821 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.609791    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs podName:e3f637a5-1cd2-41a6-98d6-ad45904012ad nodeName:}" failed. No retries permitted until 2026-10-31 12:59:18.609779324 +0000 UTC m=+10.006620698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs") pod "network-metrics-daemon-jbb9m" (UID: "e3f637a5-1cd2-41a6-98d6-ad45904012ad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:14.609991 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.609831    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:14.609991 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.609850    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret podName:97d66a46-02bb-4220-a78c-850e615072ca nodeName:}" failed. No retries permitted until 2026-10-31 12:59:18.609843842 +0000 UTC m=+10.006685216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret") pod "global-pull-secret-syncer-c6hpz" (UID: "97d66a46-02bb-4220-a78c-850e615072ca") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:14.812036 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:14.811973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:14.819492 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.819461    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:59:14.819640 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.819501    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:59:14.819640 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.819519    2576 projected.go:196] Error preparing data for projected volume kube-api-access-t6db7 for pod openshift-network-diagnostics/network-check-target-r4bj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:14.819640 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:14.819595    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7 podName:3c98a803-2cc2-4806-8a14-b05c31ee7f03 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:18.819575129 +0000 UTC m=+10.216416510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6db7" (UniqueName: "kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7") pod "network-check-target-r4bj6" (UID: "3c98a803-2cc2-4806-8a14-b05c31ee7f03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:15.091469 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:15.091380    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:15.091930 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:15.091526    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:16.089312 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:16.089281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:16.089494 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:16.089406    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:16.089915 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:16.089812    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:16.090042 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:16.089937    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:16.266721 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:16.266611    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-phbg5" event={"ID":"686b38d0-13b9-4d2c-a1cf-5b8053804350","Type":"ContainerStarted","Data":"0f5fcae9e8ca19212422bee5332251b45b21eac7a8d475f36ff8d63bea4100a6"}
Oct 31 12:59:16.285470 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:16.285049    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal" podStartSLOduration=6.2850297059999995 podStartE2EDuration="6.285029706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:14.296658127 +0000 UTC m=+5.693499504" watchObservedRunningTime="2026-10-31 12:59:16.285029706 +0000 UTC m=+7.681871090"
Oct 31 12:59:17.091913 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:17.091876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:17.092076 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:17.092000    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:17.272746 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:17.272641    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lvtzl" event={"ID":"e22c3df7-5789-4439-ba22-3a7d2b30c3fa","Type":"ContainerStarted","Data":"1d553032a34eec2af291cf66037c90db8af65132f6c361552538226e44ca5a38"}
Oct 31 12:59:17.276519 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:17.276340    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb" event={"ID":"5b16b0cb-c872-4fee-8a2a-83adb2777cc6","Type":"ContainerStarted","Data":"2352314e1d339a4bf085b195a5ed4f0e26c9005cbfe0d8644d535d430d9c1d37"}
Oct 31 12:59:17.291292 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:17.291236    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-phbg5" podStartSLOduration=4.366654531 podStartE2EDuration="8.291217309s" totalImagesPullingTime="3.924562778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:16.286321554 +0000 UTC m=+7.683162939" watchObservedRunningTime="2026-10-31 12:59:17.291217309 +0000 UTC m=+8.688058694"
Oct 31 12:59:18.089691 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:18.089616    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:18.089842 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.089786    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:18.090187 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:18.090164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:18.090278 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.090262    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:18.649438 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:18.649390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:18.649851 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:18.649455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:18.649851 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.649586    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:18.649851 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.649648    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret podName:97d66a46-02bb-4220-a78c-850e615072ca nodeName:}" failed. No retries permitted until 2026-10-31 12:59:26.649628393 +0000 UTC m=+18.046469768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret") pod "global-pull-secret-syncer-c6hpz" (UID: "97d66a46-02bb-4220-a78c-850e615072ca") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:18.650137 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.650048    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:18.650137 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.650111    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs podName:e3f637a5-1cd2-41a6-98d6-ad45904012ad nodeName:}" failed. No retries permitted until 2026-10-31 12:59:26.65009404 +0000 UTC m=+18.046935415 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs") pod "network-metrics-daemon-jbb9m" (UID: "e3f637a5-1cd2-41a6-98d6-ad45904012ad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:18.811807 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:18.811751    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 12:59:18.850367 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:18.850336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:18.850524 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.850498    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:59:18.850524 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.850512    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:59:18.850524 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.850521    2576 projected.go:196] Error preparing data for projected volume kube-api-access-t6db7 for pod openshift-network-diagnostics/network-check-target-r4bj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:18.850631 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:18.850562    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7 podName:3c98a803-2cc2-4806-8a14-b05c31ee7f03 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:26.850549852 +0000 UTC m=+18.247391212 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6db7" (UniqueName: "kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7") pod "network-check-target-r4bj6" (UID: "3c98a803-2cc2-4806-8a14-b05c31ee7f03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:19.035928 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.034960    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T12:59:18.811770815Z","UUID":"871c5204-0f79-4243-9a44-30ddacf4364e","Handler":null,"Name":"","Endpoint":""}
Oct 31 12:59:19.037242 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.036673    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 12:59:19.037242 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.036698    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 12:59:19.088881 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.088856    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:19.090128 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:19.089834    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:19.129484 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.129457    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:19.288735 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.285672    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7pzvw" event={"ID":"5337ff71-42fc-4f7d-a92f-67e13f9c0a00","Type":"ContainerStarted","Data":"42b6df3bd72b06250f4165d19e112dc93c97ffdec2eeb12f123992a902ed5fbd"}
Oct 31 12:59:19.299846 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.299161    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" event={"ID":"5510e6da-dd62-4c2f-b73e-75ca74cad962","Type":"ContainerStarted","Data":"b08d62ab1647aed24a17736fee085ea9a41a5b50da9faf49cc1b52f974d1392a"}
Oct 31 12:59:19.303312 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.303183    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ld8vz" event={"ID":"448c900f-a9c5-4df1-b49e-e76269934da4","Type":"ContainerStarted","Data":"4e77f0be73c9f067d3d5cddf086bf2379bc653694f38ac99377f5f9814a9747b"}
Oct 31 12:59:19.309118 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.308361    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb" event={"ID":"5b16b0cb-c872-4fee-8a2a-83adb2777cc6","Type":"ContainerStarted","Data":"7a562b01b54e0c662eb7945b7194fbbcacd135ca23bf94f28fa5ef1216ecbf99"}
Oct 31 12:59:19.318900 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.317165    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lvtzl" podStartSLOduration=5.3313426360000005 podStartE2EDuration="10.317147681s" totalImagesPullingTime="4.985805045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:17.292154743 +0000 UTC m=+8.688996125" watchObservedRunningTime="2026-10-31 12:59:19.317147681 +0000 UTC m=+10.713989064"
Oct 31 12:59:19.345777 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.345158    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7pzvw" podStartSLOduration=2.665204308 podStartE2EDuration="10.3451451s" totalImagesPullingTime="7.679940792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:19.317756605 +0000 UTC m=+10.714597989" watchObservedRunningTime="2026-10-31 12:59:19.3451451 +0000 UTC m=+10.741986482"
Oct 31 12:59:19.386309 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.386249    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ld8vz" podStartSLOduration=2.732639355 podStartE2EDuration="10.386233202s" totalImagesPullingTime="7.653593847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:19.345654218 +0000 UTC m=+10.742495618" watchObservedRunningTime="2026-10-31 12:59:19.386233202 +0000 UTC m=+10.783074589"
Oct 31 12:59:19.898723 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.898680    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:19.899460 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:19.899424    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:20.089176 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:20.089141    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:20.089358 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:20.089191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:20.089358 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:20.089298    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:20.089434 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:20.089413    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:20.314120 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:20.314083    2576 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="b08d62ab1647aed24a17736fee085ea9a41a5b50da9faf49cc1b52f974d1392a" exitCode=0
Oct 31 12:59:20.314285 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:20.314151    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" event={"ID":"5510e6da-dd62-4c2f-b73e-75ca74cad962","Type":"ContainerDied","Data":"b08d62ab1647aed24a17736fee085ea9a41a5b50da9faf49cc1b52f974d1392a"}
Oct 31 12:59:20.317643 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:20.317619    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb" event={"ID":"5b16b0cb-c872-4fee-8a2a-83adb2777cc6","Type":"ContainerStarted","Data":"59bed7f8c7414eb162f9faf5574269add6ba2e2d7bd4a81b09910edd71410f68"}
Oct 31 12:59:20.318729 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:20.318300    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-phbg5"
Oct 31 12:59:20.380958 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:20.380862    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb" podStartSLOduration=3.291130261 podStartE2EDuration="11.380848082s" totalImagesPullingTime="8.089717821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:20.380409555 +0000 UTC m=+11.777250941" watchObservedRunningTime="2026-10-31 12:59:20.380848082 +0000 UTC m=+11.777689465"
Oct 31 12:59:21.089932 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:21.089901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:21.090316 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:21.090044    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:21.320399 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:21.320368    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4p8sf" event={"ID":"5c4a59dc-f603-4d34-a82a-23813a1b86b6","Type":"ContainerStarted","Data":"da1f94e152916e54289252df459b6125e228bb4a637156be8bdad4c4f6aaec9d"}
Oct 31 12:59:21.339572 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:21.339523    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4p8sf" podStartSLOduration=4.367793828 podStartE2EDuration="12.339511284s" totalImagesPullingTime="7.971717456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:21.339181654 +0000 UTC m=+12.736023027" watchObservedRunningTime="2026-10-31 12:59:21.339511284 +0000 UTC m=+12.736352663"
Oct 31 12:59:22.089493 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:22.089454    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:22.089639 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:22.089564    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:22.089639 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:22.089622    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:22.089749 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:22.089731    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:22.324855 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:22.324818    2576 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="3341e8d2059416a577e4c6f9c96068a94bc1f0e2afc0728b6726b26a3f129e7b" exitCode=0
Oct 31 12:59:22.325604 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:22.324909    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" event={"ID":"5510e6da-dd62-4c2f-b73e-75ca74cad962","Type":"ContainerDied","Data":"3341e8d2059416a577e4c6f9c96068a94bc1f0e2afc0728b6726b26a3f129e7b"}
Oct 31 12:59:22.326880 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:22.326855    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5pwr7" event={"ID":"1e7e4656-607a-4995-a250-a3bda30be0c1","Type":"ContainerStarted","Data":"6639d97350141122a6d3fd68ab19f3b6504e92393bc9298695070a26d0cae4b0"}
Oct 31 12:59:22.372410 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:22.372306    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5pwr7" podStartSLOduration=3.058191653 podStartE2EDuration="13.372288204s" totalImagesPullingTime="10.314096551s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:22.371819079 +0000 UTC m=+13.768660463" watchObservedRunningTime="2026-10-31 12:59:22.372288204 +0000 UTC m=+13.769129586"
Oct 31 12:59:23.089274 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:23.089240    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:23.089455 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:23.089375    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:23.332256 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:23.332183    2576 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="62516776e5fc5ee27cc7fda29f0f5744e7f5eb5920e57fa55fe7f71914814d06" exitCode=0
Oct 31 12:59:23.332798 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:23.332266    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" event={"ID":"5510e6da-dd62-4c2f-b73e-75ca74cad962","Type":"ContainerDied","Data":"62516776e5fc5ee27cc7fda29f0f5744e7f5eb5920e57fa55fe7f71914814d06"}
Oct 31 12:59:24.089359 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:24.089167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:24.089525 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:24.089360    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:24.089525 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:24.089459    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:24.089954 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:24.089896    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:25.091420 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:25.089341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:25.091420 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:25.089474    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:25.339447 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:25.339361    2576 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="dd920d67b57f04fa7777d1bb8b6c2c4f003e1091571a5bc064b7b0f003306d0c" exitCode=0
Oct 31 12:59:25.339447 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:25.339422    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" event={"ID":"5510e6da-dd62-4c2f-b73e-75ca74cad962","Type":"ContainerDied","Data":"dd920d67b57f04fa7777d1bb8b6c2c4f003e1091571a5bc064b7b0f003306d0c"}
Oct 31 12:59:26.089395 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:26.089351    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:26.089395 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:26.089384    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:26.089661 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.089483    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:26.089743 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.089657    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:26.345974 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:26.345373    2576 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="fd9b35c46c223a485ac1f35f2322bd37f78e4e3a48ff042b6f49ba9d00b635d6" exitCode=0
Oct 31 12:59:26.345974 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:26.345430    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" event={"ID":"5510e6da-dd62-4c2f-b73e-75ca74cad962","Type":"ContainerDied","Data":"fd9b35c46c223a485ac1f35f2322bd37f78e4e3a48ff042b6f49ba9d00b635d6"}
Oct 31 12:59:26.714543 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:26.714504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:26.714736 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:26.714569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:26.714736 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.714639    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:26.714736 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.714693    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:26.714736 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.714714    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs podName:e3f637a5-1cd2-41a6-98d6-ad45904012ad nodeName:}" failed. No retries permitted until 2026-10-31 12:59:42.714686434 +0000 UTC m=+34.111527795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs") pod "network-metrics-daemon-jbb9m" (UID: "e3f637a5-1cd2-41a6-98d6-ad45904012ad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:26.714922 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.714768    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret podName:97d66a46-02bb-4220-a78c-850e615072ca nodeName:}" failed. No retries permitted until 2026-10-31 12:59:42.714751539 +0000 UTC m=+34.111592914 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret") pod "global-pull-secret-syncer-c6hpz" (UID: "97d66a46-02bb-4220-a78c-850e615072ca") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:26.916343 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:26.916310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:26.916503 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.916462    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:59:26.916503 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.916484    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:59:26.916503 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.916493    2576 projected.go:196] Error preparing data for projected volume kube-api-access-t6db7 for pod openshift-network-diagnostics/network-check-target-r4bj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:26.916617 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:26.916546    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7 podName:3c98a803-2cc2-4806-8a14-b05c31ee7f03 nodeName:}" failed. No retries permitted until 2026-10-31 12:59:42.916529943 +0000 UTC m=+34.313371308 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6db7" (UniqueName: "kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7") pod "network-check-target-r4bj6" (UID: "3c98a803-2cc2-4806-8a14-b05c31ee7f03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:27.090012 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:27.089929    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:27.090167 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:27.090073    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:27.351698 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:27.351611    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" event={"ID":"5510e6da-dd62-4c2f-b73e-75ca74cad962","Type":"ContainerStarted","Data":"84595c612dfff870351b5cb7b1e291faaa7d20be4f5273367c9afb462e28bede"}
Oct 31 12:59:27.380268 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:27.380215    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6rkkf" podStartSLOduration=7.238909185 podStartE2EDuration="18.380200618s" totalImagesPullingTime="11.141291433s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:27.378547998 +0000 UTC m=+18.775389400" watchObservedRunningTime="2026-10-31 12:59:27.380200618 +0000 UTC m=+18.777042000"
Oct 31 12:59:28.090032 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:28.089848    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:28.090200 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:28.089870    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:28.090200 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:28.090116    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:28.090200 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:28.090168    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:28.357000 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:28.356920    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" event={"ID":"87328843-fa88-4d5d-9904-5a3036d5bb10","Type":"ContainerStarted","Data":"282e68ea7853dc22759fcd79fdb8435cbcf971babc7e5b8d9dabcc41150ce6dd"}
Oct 31 12:59:28.357000 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:28.356986    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" event={"ID":"87328843-fa88-4d5d-9904-5a3036d5bb10","Type":"ContainerStarted","Data":"512b2882cfff61011c47c31a44916258945961b99abdccf1ffec5321631f3442"}
Oct 31 12:59:28.357360 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:28.357017    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" event={"ID":"87328843-fa88-4d5d-9904-5a3036d5bb10","Type":"ContainerStarted","Data":"a782f3bfa8a0807910f066d9df03ff3fe8696ec3cd1969eeadf774e29ce4d9e9"}
Oct 31 12:59:28.357360 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:28.357044    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" event={"ID":"87328843-fa88-4d5d-9904-5a3036d5bb10","Type":"ContainerStarted","Data":"db6166a6890ca82e14bf66a312604c5ae2a9c440f4897dde79b1097edb18ee70"}
Oct 31 12:59:28.357360 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:28.357071    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" event={"ID":"87328843-fa88-4d5d-9904-5a3036d5bb10","Type":"ContainerStarted","Data":"ad5124bbc1502342e7233cb3496baf871817744ed2254a6c53f45921cfc84559"}
Oct 31 12:59:28.357360 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:28.357096    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" event={"ID":"87328843-fa88-4d5d-9904-5a3036d5bb10","Type":"ContainerStarted","Data":"60ce23049bc07fb2d66583241430cd12d1fc8594ccf2d2d83ccc59ec22305c7f"}
Oct 31 12:59:29.090173 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:29.090140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:29.090340 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:29.090229    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:30.089236 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:30.089201    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:30.089673 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:30.089314    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:30.089673 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:30.089369    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:30.089673 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:30.089456    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:30.363688 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:30.363655    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" event={"ID":"87328843-fa88-4d5d-9904-5a3036d5bb10","Type":"ContainerStarted","Data":"60e568900e0217a10d2fd1df8c55c77699346f2149e229b045856a2b025f23c7"}
Oct 31 12:59:31.088993 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:31.088968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:31.089216 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:31.089060    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:32.089832 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:32.089799    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:32.089832 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:32.089816    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:32.090424 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:32.089909    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:32.090424 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:32.090027    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:32.370724 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:32.370616    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" event={"ID":"87328843-fa88-4d5d-9904-5a3036d5bb10","Type":"ContainerStarted","Data":"299e9b1ccf3afcf0df48127a3aa18f1c5ced436db57a85dee7342545c123a8e6"}
Oct 31 12:59:32.370966 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:32.370943    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:32.371038 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:32.371020    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:32.387625 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:32.387597    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:32.387793 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:32.387698    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:32.407244 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:32.407200    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb" podStartSLOduration=7.174013727 podStartE2EDuration="23.407189352s" totalImagesPullingTime="16.233175625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 12:59:32.406770254 +0000 UTC m=+23.803611637" watchObservedRunningTime="2026-10-31 12:59:32.407189352 +0000 UTC m=+23.804030735"
Oct 31 12:59:33.089356 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:33.089325    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:33.089545 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:33.089426    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:33.372476 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:33.372390    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:59:33.916097 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:33.915760    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r4bj6"]
Oct 31 12:59:33.916249 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:33.916117    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:33.916249 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:33.916229    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:33.916619 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:33.916594    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c6hpz"]
Oct 31 12:59:33.916793 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:33.916725    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:33.916867 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:33.916810    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:33.917244 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:33.917225    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jbb9m"]
Oct 31 12:59:33.917319 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:33.917313    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:33.917402 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:33.917388    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:34.374450 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:34.374421    2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 12:59:35.089571 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:35.089534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:35.089767 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:35.089666    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:36.089909 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:36.089874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:36.089909 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:36.089902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:36.090430 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:36.089993    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:36.090430 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:36.090090    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:36.559828 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:36.559794    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 12:59:37.089841 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:37.089806    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:37.090035 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:37.089912    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:38.089138 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:38.089113    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:38.089138 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:38.089119    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:38.089325 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:38.089222    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:38.089371 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:38.089356    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:38.898948 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:38.898909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-phbg5_686b38d0-13b9-4d2c-a1cf-5b8053804350/konnectivity-agent/0.log" limit=1024
Oct 31 12:59:39.042375 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:39.042340    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-phbg5_686b38d0-13b9-4d2c-a1cf-5b8053804350/konnectivity-agent/0.log" limit=1024
Oct 31 12:59:39.090104 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:39.090071    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:39.090261 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:39.090156    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:39.292544 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:39.292507    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-phbg5_686b38d0-13b9-4d2c-a1cf-5b8053804350/konnectivity-agent/0.log" limit=1024
Oct 31 12:59:40.089657 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:40.089630    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:40.090067 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:40.089766    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:40.090067 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:40.089822    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:40.090067 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:40.089935    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:41.089628 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:41.089597    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:41.089846 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:41.089746    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:42.089540 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:42.089509    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:42.089741 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:42.089513    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:42.089741 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.089606    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:42.089832 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.089743    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:42.727583 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:42.727552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:42.727942 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:42.727593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:42.727942 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.727698    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:42.727942 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.727698    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:42.727942 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.727770    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs podName:e3f637a5-1cd2-41a6-98d6-ad45904012ad nodeName:}" failed. No retries permitted until 2026-10-31 13:00:14.727753781 +0000 UTC m=+66.124595141 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs") pod "network-metrics-daemon-jbb9m" (UID: "e3f637a5-1cd2-41a6-98d6-ad45904012ad") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 12:59:42.727942 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.727783    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret podName:97d66a46-02bb-4220-a78c-850e615072ca nodeName:}" failed. No retries permitted until 2026-10-31 13:00:14.727777169 +0000 UTC m=+66.124618529 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret") pod "global-pull-secret-syncer-c6hpz" (UID: "97d66a46-02bb-4220-a78c-850e615072ca") : object "kube-system"/"original-pull-secret" not registered
Oct 31 12:59:42.929244 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:42.929193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:42.929418 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.929335    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 12:59:42.929418 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.929349    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 12:59:42.929418 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.929360    2576 projected.go:196] Error preparing data for projected volume kube-api-access-t6db7 for pod openshift-network-diagnostics/network-check-target-r4bj6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:42.929418 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:42.929411    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7 podName:3c98a803-2cc2-4806-8a14-b05c31ee7f03 nodeName:}" failed. No retries permitted until 2026-10-31 13:00:14.929393325 +0000 UTC m=+66.326234703 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6db7" (UniqueName: "kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7") pod "network-check-target-r4bj6" (UID: "3c98a803-2cc2-4806-8a14-b05c31ee7f03") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 12:59:43.089229 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:43.089156    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:43.089388 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:43.089268    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:43.454580 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:43.454555    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ld8vz_448c900f-a9c5-4df1-b49e-e76269934da4/dns-node-resolver/0.log"
Oct 31 12:59:44.089473 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:44.089444    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:44.089941 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:44.089561    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:44.089941 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:44.089607    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:44.089941 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:44.089699    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:44.254832 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:44.254809    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lvtzl_e22c3df7-5789-4439-ba22-3a7d2b30c3fa/node-ca/0.log"
Oct 31 12:59:45.089338 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:45.089303    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:45.089532 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:45.089416    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:46.089587 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:46.089557    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:46.089983 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:46.089646    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:46.089983 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:46.089742    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:46.089983 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:46.089844    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:47.089919 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:47.089885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:47.090415 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:47.090018    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:48.089448 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:48.089409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:48.089448 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:48.089455    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:48.089737 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:48.089562    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:48.089737 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:48.089677    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:49.090378 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:49.090353    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:49.090773 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:49.090434    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:50.089022 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:50.088989    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:50.089022 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:50.089022    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:50.089244 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:50.089107    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:50.089244 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:50.089235    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:51.089906 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:51.089865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:51.090308 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:51.090009    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:52.089526 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:52.089494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:52.089683 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:52.089595    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:52.089683 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:52.089623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:52.089683 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:52.089676    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:53.089514 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:53.089482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:53.089924 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:53.089607    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:54.089766 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:54.089727    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:54.090231 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:54.089728    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:54.090231 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:54.089882    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:54.090231 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:54.089902    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:55.092362 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:55.092331    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:55.092822 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:55.092447    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:56.089193 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:56.089148    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:56.089193 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:56.089184    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:56.089413 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:56.089265    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:56.089413 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:56.089379    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:57.089352 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:57.089315    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:57.089780 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:57.089422    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-c6hpz" podUID="97d66a46-02bb-4220-a78c-850e615072ca"
Oct 31 12:59:58.089147 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.089117    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 12:59:58.089307 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:58.089212    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-r4bj6" podUID="3c98a803-2cc2-4806-8a14-b05c31ee7f03"
Oct 31 12:59:58.089307 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.089249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 12:59:58.089382 ip-10-0-141-83 kubenswrapper[2576]: E1031 12:59:58.089309    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbb9m" podUID="e3f637a5-1cd2-41a6-98d6-ad45904012ad"
Oct 31 12:59:58.869161 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.869133    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeReady"
Oct 31 12:59:58.869323 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.869236    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 12:59:58.919495 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.919466    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wwg8z"]
Oct 31 12:59:58.921158 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.921143    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:58.925343 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.925320    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2798k"]
Oct 31 12:59:58.926843 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.926825    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 12:59:58.926992 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.926966    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 12:59:58.927319 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.927305    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftfm5\""
Oct 31 12:59:58.927385 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.927368    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 12:59:58.938469 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.938444    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wwg8z"]
Oct 31 12:59:58.938982 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.938966    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2798k"]
Oct 31 12:59:58.943488 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.943468    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nkxs6"]
Oct 31 12:59:58.946928 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.946906    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:58.964029 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:58.964002    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nkxs6"]
Oct 31 12:59:59.027359 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.027326    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 12:59:59.027535 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.027479    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 12:59:59.027535 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.027522    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 12:59:59.027642 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.027536    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 12:59:59.027642 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.027544    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 12:59:59.027642 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.027572    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 12:59:59.027642 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.027584    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gth9v\""
Oct 31 12:59:59.027642 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.027590    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 12:59:59.027642 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.027596    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qpv6m\""
Oct 31 12:59:59.036839 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.036819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f24485c-3402-4cef-84eb-6ea5dd63be64-cert\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 12:59:59.036886 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.036871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc748e5-08b1-4ffa-900c-09730777cc8d-tmp-dir\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.036924 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.036898    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dc748e5-08b1-4ffa-900c-09730777cc8d-metrics-tls\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.037002 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.036985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2frp7\" (UniqueName: \"kubernetes.io/projected/7f24485c-3402-4cef-84eb-6ea5dd63be64-kube-api-access-2frp7\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 12:59:59.037049 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.037037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc748e5-08b1-4ffa-900c-09730777cc8d-config-volume\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.037081 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.037056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjtw4\" (UniqueName: \"kubernetes.io/projected/3dc748e5-08b1-4ffa-900c-09730777cc8d-kube-api-access-xjtw4\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.092412 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.092382    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 12:59:59.096519 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.096502    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 12:59:59.096574 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.096508    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lt4jt\""
Oct 31 12:59:59.137551 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc748e5-08b1-4ffa-900c-09730777cc8d-config-volume\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.137551 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtw4\" (UniqueName: \"kubernetes.io/projected/3dc748e5-08b1-4ffa-900c-09730777cc8d-kube-api-access-xjtw4\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.137551 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f24485c-3402-4cef-84eb-6ea5dd63be64-cert\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 12:59:59.137551 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-data-volume\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.137861 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137747    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-crio-socket\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.137861 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc748e5-08b1-4ffa-900c-09730777cc8d-tmp-dir\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.137962 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.137962 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dc748e5-08b1-4ffa-900c-09730777cc8d-metrics-tls\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.137962 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137946    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsq8j\" (UniqueName: \"kubernetes.io/projected/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-api-access-rsq8j\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.138087 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.137979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2frp7\" (UniqueName: \"kubernetes.io/projected/7f24485c-3402-4cef-84eb-6ea5dd63be64-kube-api-access-2frp7\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 12:59:59.138087 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.138001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.138182 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.138111    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc748e5-08b1-4ffa-900c-09730777cc8d-tmp-dir\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.138182 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.138127    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc748e5-08b1-4ffa-900c-09730777cc8d-config-volume\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.141295 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.141262    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dc748e5-08b1-4ffa-900c-09730777cc8d-metrics-tls\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.141389 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.141302    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f24485c-3402-4cef-84eb-6ea5dd63be64-cert\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 12:59:59.151976 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.151951    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtw4\" (UniqueName: \"kubernetes.io/projected/3dc748e5-08b1-4ffa-900c-09730777cc8d-kube-api-access-xjtw4\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.152408 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.152391    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2frp7\" (UniqueName: \"kubernetes.io/projected/7f24485c-3402-4cef-84eb-6ea5dd63be64-kube-api-access-2frp7\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 12:59:59.232052 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.232019    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wwg8z"
Oct 31 12:59:59.237153 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.237131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 12:59:59.238490 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.238464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.238560 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.238541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-data-volume\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.238628 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.238581    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-crio-socket\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.238628 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.238615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.238728 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.238642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsq8j\" (UniqueName: \"kubernetes.io/projected/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-api-access-rsq8j\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.238837 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.238816    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-crio-socket\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.239157 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.239130    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-data-volume\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.239235 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.239161    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.241534 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.241510    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.252079 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.252056    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsq8j\" (UniqueName: \"kubernetes.io/projected/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-api-access-rsq8j\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.259739 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.259653    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 12:59:59.525798 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.525768    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2798k"]
Oct 31 12:59:59.528165 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.528139    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wwg8z"]
Oct 31 12:59:59.530998 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:59.530944    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dc748e5_08b1_4ffa_900c_09730777cc8d.slice/crio-8d67c669e0d954f3b9e96cecfadc7a89b4aa3c07d161493e489c50661e2819f8 WatchSource:0}: Error finding container 8d67c669e0d954f3b9e96cecfadc7a89b4aa3c07d161493e489c50661e2819f8: Status 404 returned error can't find the container with id 8d67c669e0d954f3b9e96cecfadc7a89b4aa3c07d161493e489c50661e2819f8
Oct 31 12:59:59.563838 ip-10-0-141-83 kubenswrapper[2576]: I1031 12:59:59.563817    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nkxs6"]
Oct 31 12:59:59.566806 ip-10-0-141-83 kubenswrapper[2576]: W1031 12:59:59.566776    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3fbffa_ba33_46db_9bdf_fcdf578955f6.slice/crio-7b77996449b59248e4f4dcbe8b045bc945865682ed50288dd5938055bb184253 WatchSource:0}: Error finding container 7b77996449b59248e4f4dcbe8b045bc945865682ed50288dd5938055bb184253: Status 404 returned error can't find the container with id 7b77996449b59248e4f4dcbe8b045bc945865682ed50288dd5938055bb184253
Oct 31 13:00:00.089016 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.088977    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 13:00:00.089212 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.089145    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 13:00:00.094323 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.094296    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 13:00:00.094694 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.094346    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 13:00:00.095437 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.095414    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g22bl\""
Oct 31 13:00:00.095553 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.095449    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 13:00:00.095553 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.095478    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lsjfn\""
Oct 31 13:00:00.432457 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.432427    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2798k" event={"ID":"7f24485c-3402-4cef-84eb-6ea5dd63be64","Type":"ContainerStarted","Data":"63a978d5f32e285b86837234e75a0ceb01f2c6fda51fd8813ca0e2b7aa3db2e5"}
Oct 31 13:00:00.434655 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.434626    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nkxs6" event={"ID":"8b3fbffa-ba33-46db-9bdf-fcdf578955f6","Type":"ContainerStarted","Data":"1cd277f51dc6616f32d19e540d17d213820b6e36cf74bfb0210cacf9c67ab9be"}
Oct 31 13:00:00.434791 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.434677    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nkxs6" event={"ID":"8b3fbffa-ba33-46db-9bdf-fcdf578955f6","Type":"ContainerStarted","Data":"7b77996449b59248e4f4dcbe8b045bc945865682ed50288dd5938055bb184253"}
Oct 31 13:00:00.436257 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:00.436224    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wwg8z" event={"ID":"3dc748e5-08b1-4ffa-900c-09730777cc8d","Type":"ContainerStarted","Data":"8d67c669e0d954f3b9e96cecfadc7a89b4aa3c07d161493e489c50661e2819f8"}
Oct 31 13:00:01.442303 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.442263    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nkxs6" event={"ID":"8b3fbffa-ba33-46db-9bdf-fcdf578955f6","Type":"ContainerStarted","Data":"46f8bac5eda5da40113df717cbb66930766eeaa6de70a30d5c60dcabd69294bb"}
Oct 31 13:00:01.466603 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.466566    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"]
Oct 31 13:00:01.468835 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.468814    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.477370 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.477348    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 13:00:01.477477 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.477415    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 13:00:01.478465 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.478433    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-d9gnb\""
Oct 31 13:00:01.478553 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.478518    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 13:00:01.478620 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.478612    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 13:00:01.478692 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.478679    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 13:00:01.484150 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.484132    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"]
Oct 31 13:00:01.656453 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.656409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.656634 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.656460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a81cd320-aea5-4bbf-9a40-f83f814180b5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.656634 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.656560    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.656634 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.656586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rxv\" (UniqueName: \"kubernetes.io/projected/a81cd320-aea5-4bbf-9a40-f83f814180b5-kube-api-access-w8rxv\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.757382 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.757288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.757382 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.757332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rxv\" (UniqueName: \"kubernetes.io/projected/a81cd320-aea5-4bbf-9a40-f83f814180b5-kube-api-access-w8rxv\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.757869 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.757831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.758008 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.757890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a81cd320-aea5-4bbf-9a40-f83f814180b5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.758716 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.758673    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a81cd320-aea5-4bbf-9a40-f83f814180b5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.761000 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.760449    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.761602 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.761580    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.771688 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.771659    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rxv\" (UniqueName: \"kubernetes.io/projected/a81cd320-aea5-4bbf-9a40-f83f814180b5-kube-api-access-w8rxv\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:01.782347 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:01.782323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:02.066464 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.066438    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"]
Oct 31 13:00:02.070065 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:02.070035    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda81cd320_aea5_4bbf_9a40_f83f814180b5.slice/crio-0086b7a63de0798c31fadaf9c57d093beab2e50bbe256195f6cf1552a95c9f59 WatchSource:0}: Error finding container 0086b7a63de0798c31fadaf9c57d093beab2e50bbe256195f6cf1552a95c9f59: Status 404 returned error can't find the container with id 0086b7a63de0798c31fadaf9c57d093beab2e50bbe256195f6cf1552a95c9f59
Oct 31 13:00:02.448304 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.448188    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nkxs6" event={"ID":"8b3fbffa-ba33-46db-9bdf-fcdf578955f6","Type":"ContainerStarted","Data":"610e05b1b0dedf444b2c6bbeed3a5cae0fd90621dcea3a54066cbc6947601cfc"}
Oct 31 13:00:02.450017 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.449991    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9" event={"ID":"a81cd320-aea5-4bbf-9a40-f83f814180b5","Type":"ContainerStarted","Data":"0086b7a63de0798c31fadaf9c57d093beab2e50bbe256195f6cf1552a95c9f59"}
Oct 31 13:00:02.451874 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.451835    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2798k" event={"ID":"7f24485c-3402-4cef-84eb-6ea5dd63be64","Type":"ContainerStarted","Data":"8cac3262e3e1ed559e3240348d894c523fd04ade554f698c8540437b6ecd0675"}
Oct 31 13:00:02.472457 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.472412    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nkxs6" podStartSLOduration=2.126822732 podStartE2EDuration="4.472400531s" totalImagesPullingTime="2.345577799s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 12:59:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:02.471073609 +0000 UTC m=+53.867914993" watchObservedRunningTime="2026-10-31 13:00:02.472400531 +0000 UTC m=+53.869241892"
Oct 31 13:00:02.490106 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.490049    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2798k" podStartSLOduration=1.7289876849999999 podStartE2EDuration="4.490035739s" totalImagesPullingTime="2.761048054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:02.488884667 +0000 UTC m=+53.885726050" watchObservedRunningTime="2026-10-31 13:00:02.490035739 +0000 UTC m=+53.886877121"
Oct 31 13:00:02.638830 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.637904    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f4dfdf7b8-7hlkd"]
Oct 31 13:00:02.640739 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.640718    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.645662 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.645641    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 13:00:02.645821 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.645804    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 13:00:02.645821 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.645813    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 13:00:02.645961 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.645870    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 13:00:02.648864 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.648846    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7zf52\""
Oct 31 13:00:02.658097 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.658075    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4dfdf7b8-7hlkd"]
Oct 31 13:00:02.666638 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.666619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprzm\" (UniqueName: \"kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.666754 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.666645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.666754 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.666662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.666754 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.666737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.666914 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.666839    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.666914 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.666889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.666989 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.666974    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.746130 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.746107    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 13:00:02.746130 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.746118    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 13:00:02.746130 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.746113    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 13:00:02.767595 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.767563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.767698 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.767619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprzm\" (UniqueName: \"kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.767768 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.767694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.767768 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.767759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.767877 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.767808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.767877 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.767860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.767975 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.767901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.767975 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.767962    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.768300 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.768278    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.768591 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.768568    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.768725 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.768687    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.770222 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.770197    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.770580 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.770558    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.782802 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.782775    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprzm\" (UniqueName: \"kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:02.962525 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:02.962484    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:03.249540 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:03.249469    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4dfdf7b8-7hlkd"]
Oct 31 13:00:03.253044 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:03.253016    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode505bf1f_b560_4226_9a6e_fad97121cbfe.slice/crio-1b45337793c79920fff6e52e54fb47fb4d48eb004fa6b5c94a75d0410e99a507 WatchSource:0}: Error finding container 1b45337793c79920fff6e52e54fb47fb4d48eb004fa6b5c94a75d0410e99a507: Status 404 returned error can't find the container with id 1b45337793c79920fff6e52e54fb47fb4d48eb004fa6b5c94a75d0410e99a507
Oct 31 13:00:03.455535 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:03.455494    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4dfdf7b8-7hlkd" event={"ID":"e505bf1f-b560-4226-9a6e-fad97121cbfe","Type":"ContainerStarted","Data":"1b45337793c79920fff6e52e54fb47fb4d48eb004fa6b5c94a75d0410e99a507"}
Oct 31 13:00:03.457553 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:03.457527    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wwg8z" event={"ID":"3dc748e5-08b1-4ffa-900c-09730777cc8d","Type":"ContainerStarted","Data":"7452d21710d620894a486f5e58065a5c2fb3e5351bc05589d6e6c773da9aeca2"}
Oct 31 13:00:03.457677 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:03.457580    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wwg8z" event={"ID":"3dc748e5-08b1-4ffa-900c-09730777cc8d","Type":"ContainerStarted","Data":"bd5e541573cfd286a1ac45ae9881f6387a3585a60f857893bcf2ceaf84aa5e89"}
Oct 31 13:00:03.478724 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:03.478650    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-wwg8z" podStartSLOduration=2.415185909 podStartE2EDuration="5.478631798s" totalImagesPullingTime="3.063445889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:03.478375921 +0000 UTC m=+54.875217330" watchObservedRunningTime="2026-10-31 13:00:03.478631798 +0000 UTC m=+54.875473182"
Oct 31 13:00:04.465909 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:04.465788    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9" event={"ID":"a81cd320-aea5-4bbf-9a40-f83f814180b5","Type":"ContainerStarted","Data":"b0e13dd90cfad6acc83ea062506b374e5e3b6fbce47c8b3115db48a0be8643c8"}
Oct 31 13:00:04.466452 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:04.466025    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9" event={"ID":"a81cd320-aea5-4bbf-9a40-f83f814180b5","Type":"ContainerStarted","Data":"c06dd5bd7a5c59843dd258efdd5bcacafdce321d5d5b0f786760e57c8874fb2f"}
Oct 31 13:00:04.466452 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:04.466308    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:04.486385 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:04.486328    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9" podStartSLOduration=2.047032196 podStartE2EDuration="3.486313073s" totalImagesPullingTime="1.439280877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:04.486312217 +0000 UTC m=+55.883153601" watchObservedRunningTime="2026-10-31 13:00:04.486313073 +0000 UTC m=+55.883154458"
Oct 31 13:00:06.572694 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.572657    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:06.850366 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.850286    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"]
Oct 31 13:00:06.856146 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.856119    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:06.873164 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.867591    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 13:00:06.873164 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.867810    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 13:00:06.873164 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.867958    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-q5jtb\""
Oct 31 13:00:06.873164 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.868482    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"]
Oct 31 13:00:06.893661 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.893633    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"]
Oct 31 13:00:06.896740 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.896062    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:06.902751 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.902730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc47f7a4-649c-4529-a608-4479486a59d8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:06.902883 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.902829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:06.902883 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.902857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:06.903009 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.902883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4666\" (UniqueName: \"kubernetes.io/projected/bc47f7a4-649c-4529-a608-4479486a59d8-kube-api-access-q4666\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:06.904791 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.904772    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 13:00:06.910760 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.910732    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-srtvx"]
Oct 31 13:00:06.913375 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.913340    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"]
Oct 31 13:00:06.913862 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.913484    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:06.937076 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.937045    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 13:00:06.937264 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.937174    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 13:00:06.937436 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.937417    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c65ps\""
Oct 31 13:00:06.937522 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.937422    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 13:00:06.938350 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.938300    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 13:00:06.938350 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:06.938317    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l8r9q\""
Oct 31 13:00:07.003645 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.003616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-wtmp\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.003774 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.003660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.003774 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.003683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.003774 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.003729    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7k9\" (UniqueName: \"kubernetes.io/projected/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-api-access-zh7k9\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.003931 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.003793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-root\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.003931 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.003840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-textfile\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.003931 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.003869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.003931 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.003908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:07.004098 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57294d98-42b2-47f0-a79a-d9cf7053ff9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.004098 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-accelerators-collector-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.004098 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004090    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.004230 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.004230 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2n6k\" (UniqueName: \"kubernetes.io/projected/3814f51b-f30b-4f73-8061-ece6249a6edb-kube-api-access-s2n6k\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.004230 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004209    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-sys\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.004382 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc47f7a4-649c-4529-a608-4479486a59d8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:07.004382 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-tls\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.004382 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:07.004524 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4666\" (UniqueName: \"kubernetes.io/projected/bc47f7a4-649c-4529-a608-4479486a59d8-kube-api-access-q4666\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:07.004524 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-metrics-client-ca\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.004925 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.004900    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc47f7a4-649c-4529-a608-4479486a59d8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:07.005182 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.005145    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 13:00:07.006376 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.006355    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:07.006778 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.006758    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:07.017442 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.017419    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4666\" (UniqueName: \"kubernetes.io/projected/bc47f7a4-649c-4529-a608-4479486a59d8-kube-api-access-q4666\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:07.105514 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-metrics-client-ca\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.105514 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-wtmp\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.105514 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7k9\" (UniqueName: \"kubernetes.io/projected/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-api-access-zh7k9\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-root\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-textfile\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105644    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-wtmp\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105657    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-root\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105699    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57294d98-42b2-47f0-a79a-d9cf7053ff9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-accelerators-collector-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.105799 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.106228 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.106228 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2n6k\" (UniqueName: \"kubernetes.io/projected/3814f51b-f30b-4f73-8061-ece6249a6edb-kube-api-access-s2n6k\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.106228 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-sys\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.106228 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-tls\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.106228 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.105994    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-textfile\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.106451 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.106279    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-metrics-client-ca\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.106451 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.106339    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.106672 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.106612    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-sys\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.106814 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.106530    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.106876 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.106815    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57294d98-42b2-47f0-a79a-d9cf7053ff9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.107047 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.107027    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-accelerators-collector-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.108587 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.108563    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-tls\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.108719 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.108621    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.109159 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.109135    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.109327 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.109311    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.119346 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.119313    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2n6k\" (UniqueName: \"kubernetes.io/projected/3814f51b-f30b-4f73-8061-ece6249a6edb-kube-api-access-s2n6k\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.119538 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.119514    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7k9\" (UniqueName: \"kubernetes.io/projected/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-api-access-zh7k9\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.176357 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.176322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:07.212140 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.212103    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:07.228449 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.228242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:07.241267 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:07.241224    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3814f51b_f30b_4f73_8061_ece6249a6edb.slice/crio-95d115eee69c9a5b55ffee64a98d09b67db14750517756dcb358958a57836c55 WatchSource:0}: Error finding container 95d115eee69c9a5b55ffee64a98d09b67db14750517756dcb358958a57836c55: Status 404 returned error can't find the container with id 95d115eee69c9a5b55ffee64a98d09b67db14750517756dcb358958a57836c55
Oct 31 13:00:07.449036 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.449012    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"]
Oct 31 13:00:07.451346 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:07.451316    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc47f7a4_649c_4529_a608_4479486a59d8.slice/crio-01cb702c3b971ea1c290687780c60bd36906a505086d0c552910122bdaf19955 WatchSource:0}: Error finding container 01cb702c3b971ea1c290687780c60bd36906a505086d0c552910122bdaf19955: Status 404 returned error can't find the container with id 01cb702c3b971ea1c290687780c60bd36906a505086d0c552910122bdaf19955
Oct 31 13:00:07.476579 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.476545    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4dfdf7b8-7hlkd" event={"ID":"e505bf1f-b560-4226-9a6e-fad97121cbfe","Type":"ContainerStarted","Data":"795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c"}
Oct 31 13:00:07.478172 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.478146    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-srtvx" event={"ID":"3814f51b-f30b-4f73-8061-ece6249a6edb","Type":"ContainerStarted","Data":"95d115eee69c9a5b55ffee64a98d09b67db14750517756dcb358958a57836c55"}
Oct 31 13:00:07.479287 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.479264    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk" event={"ID":"bc47f7a4-649c-4529-a608-4479486a59d8","Type":"ContainerStarted","Data":"01cb702c3b971ea1c290687780c60bd36906a505086d0c552910122bdaf19955"}
Oct 31 13:00:07.494040 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.494017    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"]
Oct 31 13:00:07.496265 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:07.496236    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57294d98_42b2_47f0_a79a_d9cf7053ff9c.slice/crio-1c114446240ba720521a0ff11a76b834dd50c44d2f2dce13ccdc332332939955 WatchSource:0}: Error finding container 1c114446240ba720521a0ff11a76b834dd50c44d2f2dce13ccdc332332939955: Status 404 returned error can't find the container with id 1c114446240ba720521a0ff11a76b834dd50c44d2f2dce13ccdc332332939955
Oct 31 13:00:07.499227 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.499112    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f4dfdf7b8-7hlkd" podStartSLOduration=1.784578343 podStartE2EDuration="5.499099193s" totalImagesPullingTime="3.71452085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:07.498040137 +0000 UTC m=+58.894881520" watchObservedRunningTime="2026-10-31 13:00:07.499099193 +0000 UTC m=+58.895940576"
Oct 31 13:00:07.950147 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.950109    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 13:00:07.952828 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.952808    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:07.957845 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.957821    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 13:00:07.958369 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.958349    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 13:00:07.958480 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.958397    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 13:00:07.958480 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.958466    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 13:00:07.958636 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.958618    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 13:00:07.958748 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.958683    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 13:00:07.958834 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.958764    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jfvtg\""
Oct 31 13:00:07.960926 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.960905    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 13:00:07.960926 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.960924    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 13:00:07.961898 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.961872    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 13:00:07.973054 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:07.973024    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 13:00:08.015791 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.015757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.015947 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.015802    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.015947 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.015827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016048 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016099 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016050    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016099 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016181 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016135    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016220 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016319 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016385 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016333    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016385 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016364    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016483 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.016527 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.016515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx59n\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.117740 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.117679    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx59n\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.117898 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.117780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.117898 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.117809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.117898 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.117831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.117898 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.117885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.118088 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.117900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.118088 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.117918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.118088 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.117951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.118088 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.117980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.118088 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.118008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.118088 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.118075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.118336 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.118102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.118336 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.118138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.119734 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.119081    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.119734 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.119394    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.120902 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.120872    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.121036 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.121018    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.121183 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.121162    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.122632 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.122319    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.122632 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.122577    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.122922 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.122891    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.123296 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.123224    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.128874 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.124344    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.128874 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.127464    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.132511 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.132484    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx59n\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.133283 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.133264    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.274817 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.274772    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:08.485382 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.485340    2576 generic.go:356] "Generic (PLEG): container finished" podID="3814f51b-f30b-4f73-8061-ece6249a6edb" containerID="b0fee4f73f846221ec445f4d95b125d00ae8cd357376aee766c102ad7a9769ff" exitCode=0
Oct 31 13:00:08.485532 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.485418    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-srtvx" event={"ID":"3814f51b-f30b-4f73-8061-ece6249a6edb","Type":"ContainerDied","Data":"b0fee4f73f846221ec445f4d95b125d00ae8cd357376aee766c102ad7a9769ff"}
Oct 31 13:00:08.487761 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.487734    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk" event={"ID":"bc47f7a4-649c-4529-a608-4479486a59d8","Type":"ContainerStarted","Data":"ace42f825b1050f8e3b7fab46ea6eaa97c0753b189e1f4ddabb8bad09110321a"}
Oct 31 13:00:08.487867 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.487795    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk" event={"ID":"bc47f7a4-649c-4529-a608-4479486a59d8","Type":"ContainerStarted","Data":"d494afef7dc42db0f8466185a12f6d4fb1b4e5b0ada871241983934e4fce3d89"}
Oct 31 13:00:08.489804 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.489776    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm" event={"ID":"57294d98-42b2-47f0-a79a-d9cf7053ff9c","Type":"ContainerStarted","Data":"1c114446240ba720521a0ff11a76b834dd50c44d2f2dce13ccdc332332939955"}
Oct 31 13:00:08.583162 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.583132    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 13:00:08.586771 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:08.586687    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab4d241_bf18_44f2_a803_1de48ef9086e.slice/crio-0736cae6a049b56e1016fcb9045064f8a3cb4b2cfd3ec2fb4c7973a886d7627a WatchSource:0}: Error finding container 0736cae6a049b56e1016fcb9045064f8a3cb4b2cfd3ec2fb4c7973a886d7627a: Status 404 returned error can't find the container with id 0736cae6a049b56e1016fcb9045064f8a3cb4b2cfd3ec2fb4c7973a886d7627a
Oct 31 13:00:08.934514 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.934444    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"]
Oct 31 13:00:08.938600 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.938061    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:08.942980 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.942956    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 13:00:08.943207 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.943189    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 13:00:08.943282 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.943200    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 13:00:08.943282 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.943201    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-esg1vsh0ffr3k\""
Oct 31 13:00:08.943282 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.943240    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 13:00:08.943417 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.943280    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 13:00:08.943417 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.943291    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2fjdc\""
Oct 31 13:00:08.954656 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:08.954625    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"]
Oct 31 13:00:09.025969 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.025917    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.026113 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.025984    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-metrics-client-ca\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.026113 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.026012    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48b5c\" (UniqueName: \"kubernetes.io/projected/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-kube-api-access-48b5c\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.026113 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.026072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.026113 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.026104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-tmp\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.026307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.026132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.026307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.026172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-grpc-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.026307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.026235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.026307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.026267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.127100 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.127024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.127100 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.127072    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.127673 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.127114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.127673 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.127163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-metrics-client-ca\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.127673 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.127189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48b5c\" (UniqueName: \"kubernetes.io/projected/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-kube-api-access-48b5c\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.127673 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.127248    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.127673 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.127282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-tmp\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.127673 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.127309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.127673 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.127354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-grpc-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.132447 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.131199    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-tmp\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.133997 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.133971    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-metrics-client-ca\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.135788 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.135741    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.139350 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.139258    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-grpc-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.144274 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.144244    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.144463 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.144340    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.144762 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.144655    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.149178 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.149153    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48b5c\" (UniqueName: \"kubernetes.io/projected/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-kube-api-access-48b5c\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.150784 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.150649    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.264907 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.264863    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:09.498358 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.498267    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm" event={"ID":"57294d98-42b2-47f0-a79a-d9cf7053ff9c","Type":"ContainerStarted","Data":"5f7838f28b2ce9ed6fdc1400abeed24ab393f187261f4e8f455091dbf5310004"}
Oct 31 13:00:09.498358 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.498345    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm" event={"ID":"57294d98-42b2-47f0-a79a-d9cf7053ff9c","Type":"ContainerStarted","Data":"7e0e24c2b594522e1d7bab88ab097956d2cebc5fc5870622cbf0a668a3869a3b"}
Oct 31 13:00:09.498590 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.498378    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm" event={"ID":"57294d98-42b2-47f0-a79a-d9cf7053ff9c","Type":"ContainerStarted","Data":"dad05fc31022ef5096a71aa9da04a64fb48617588f8aacc571c5fd6207d49319"}
Oct 31 13:00:09.499766 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.499734    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerStarted","Data":"0736cae6a049b56e1016fcb9045064f8a3cb4b2cfd3ec2fb4c7973a886d7627a"}
Oct 31 13:00:09.502088 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.502058    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-srtvx" event={"ID":"3814f51b-f30b-4f73-8061-ece6249a6edb","Type":"ContainerStarted","Data":"1c798960a27d995d6225e5acaac93128df52967b4c6680f06c78e79fb4ff7500"}
Oct 31 13:00:09.502210 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.502116    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-srtvx" event={"ID":"3814f51b-f30b-4f73-8061-ece6249a6edb","Type":"ContainerStarted","Data":"43606bb1eeea522a6e558a56768680e97b4531542780eb9955c301a08e698757"}
Oct 31 13:00:09.504186 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.504159    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk" event={"ID":"bc47f7a4-649c-4529-a608-4479486a59d8","Type":"ContainerStarted","Data":"6205b48b9e2f00eb5ccf2072e9463598df8d8e29a5c31c9fb9efd664fd693fe8"}
Oct 31 13:00:09.523662 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.523597    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm" podStartSLOduration=2.057106514 podStartE2EDuration="3.523576502s" totalImagesPullingTime="1.466469988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:09.521521679 +0000 UTC m=+60.918363079" watchObservedRunningTime="2026-10-31 13:00:09.523576502 +0000 UTC m=+60.920417886"
Oct 31 13:00:09.551029 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.550866    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk" podStartSLOduration=1.968736379 podStartE2EDuration="3.550851022s" totalImagesPullingTime="1.582114643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:09.549775385 +0000 UTC m=+60.946616786" watchObservedRunningTime="2026-10-31 13:00:09.550851022 +0000 UTC m=+60.947692412"
Oct 31 13:00:09.558561 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.558531    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"]
Oct 31 13:00:09.563675 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:09.563643    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcc9522_04bc_46ec_873c_ac4d00b96ffb.slice/crio-72518e06fb622b5bdf461b87e80e3db32ad81a20fd610a0acfd2ea58fbebfc55 WatchSource:0}: Error finding container 72518e06fb622b5bdf461b87e80e3db32ad81a20fd610a0acfd2ea58fbebfc55: Status 404 returned error can't find the container with id 72518e06fb622b5bdf461b87e80e3db32ad81a20fd610a0acfd2ea58fbebfc55
Oct 31 13:00:09.578497 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:09.578445    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-srtvx" podStartSLOduration=2.460171104 podStartE2EDuration="3.578430314s" totalImagesPullingTime="1.015938502s" totalInitContainerRuntime="102.320708ms" isStatefulPod=true podCreationTimestamp="2026-10-31 13:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:09.577829268 +0000 UTC m=+60.974670652" watchObservedRunningTime="2026-10-31 13:00:09.578430314 +0000 UTC m=+60.975271739"
Oct 31 13:00:10.516413 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:10.516372    2576 generic.go:356] "Generic (PLEG): container finished" podID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerID="b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055" exitCode=0
Oct 31 13:00:10.516804 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:10.516459    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerDied","Data":"b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055"}
Oct 31 13:00:10.518243 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:10.518219    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv" event={"ID":"2bcc9522-04bc-46ec-873c-ac4d00b96ffb","Type":"ContainerStarted","Data":"72518e06fb622b5bdf461b87e80e3db32ad81a20fd610a0acfd2ea58fbebfc55"}
Oct 31 13:00:11.261128 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.261092    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"]
Oct 31 13:00:11.265514 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.264667    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.270530 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.270508    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 13:00:11.270644 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.270549    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 13:00:11.275876 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.275854    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"]
Oct 31 13:00:11.294984 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.294960    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 13:00:11.295279 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.295256    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 13:00:11.295385 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.295352    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4r7h54n5eni9s\""
Oct 31 13:00:11.295452 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.295440    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fm9ml\""
Oct 31 13:00:11.352749 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.352692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41aeee13-d6f6-4b22-b0d0-5c6a88038716-audit-log\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.352914 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.352826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-metrics-server-audit-profiles\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.352914 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.352886    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-client-certs\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.352914 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.352905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nhc\" (UniqueName: \"kubernetes.io/projected/41aeee13-d6f6-4b22-b0d0-5c6a88038716-kube-api-access-n2nhc\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.353012 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.352998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-tls\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.353044 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.353024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.353073 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.353048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-client-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.453436 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.453397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-client-certs\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.453436 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.453436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nhc\" (UniqueName: \"kubernetes.io/projected/41aeee13-d6f6-4b22-b0d0-5c6a88038716-kube-api-access-n2nhc\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.453667 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.453492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-tls\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.453667 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.453514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.453667 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.453544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-client-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.453667 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.453587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41aeee13-d6f6-4b22-b0d0-5c6a88038716-audit-log\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.453667 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.453644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-metrics-server-audit-profiles\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.454178 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.454118    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41aeee13-d6f6-4b22-b0d0-5c6a88038716-audit-log\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.454409 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.454382    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.454991 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.454964    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-metrics-server-audit-profiles\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.456454 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.456429    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-client-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.456754 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.456728    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-tls\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.456847 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.456728    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-client-certs\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.467066 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.467041    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nhc\" (UniqueName: \"kubernetes.io/projected/41aeee13-d6f6-4b22-b0d0-5c6a88038716-kube-api-access-n2nhc\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.579631 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.579522    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:11.638838 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.638685    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"]
Oct 31 13:00:11.643731 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.642395    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:11.649565 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.648956    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lz75d\""
Oct 31 13:00:11.654120 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.652315    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"]
Oct 31 13:00:11.748814 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.748789    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 13:00:11.758185 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.758149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be0506ab-10a8-4158-b7e1-62ded5ce9f11-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:11.758373 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.758346    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nwdg\" (UniqueName: \"kubernetes.io/projected/be0506ab-10a8-4158-b7e1-62ded5ce9f11-kube-api-access-9nwdg\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:11.859771 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.859665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwdg\" (UniqueName: \"kubernetes.io/projected/be0506ab-10a8-4158-b7e1-62ded5ce9f11-kube-api-access-9nwdg\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:11.859929 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.859773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be0506ab-10a8-4158-b7e1-62ded5ce9f11-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:11.862508 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.862485    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be0506ab-10a8-4158-b7e1-62ded5ce9f11-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:11.865500 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.865472    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"]
Oct 31 13:00:11.868891 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:11.868867    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41aeee13_d6f6_4b22_b0d0_5c6a88038716.slice/crio-7c1d4b8931b94666de3faeb88062c0338b6e4dcf70d31359d5ba60fd97912843 WatchSource:0}: Error finding container 7c1d4b8931b94666de3faeb88062c0338b6e4dcf70d31359d5ba60fd97912843: Status 404 returned error can't find the container with id 7c1d4b8931b94666de3faeb88062c0338b6e4dcf70d31359d5ba60fd97912843
Oct 31 13:00:11.873518 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.873470    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwdg\" (UniqueName: \"kubernetes.io/projected/be0506ab-10a8-4158-b7e1-62ded5ce9f11-kube-api-access-9nwdg\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:11.971158 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:11.971129    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:12.256963 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.256942    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"]
Oct 31 13:00:12.263575 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:12.263545    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe0506ab_10a8_4158_b7e1_62ded5ce9f11.slice/crio-f7eb57b0fd124d00b8fd47f901e74808fc8c745743269c37aa8fc4dc38cc0084 WatchSource:0}: Error finding container f7eb57b0fd124d00b8fd47f901e74808fc8c745743269c37aa8fc4dc38cc0084: Status 404 returned error can't find the container with id f7eb57b0fd124d00b8fd47f901e74808fc8c745743269c37aa8fc4dc38cc0084
Oct 31 13:00:12.420368 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.419222    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-64cd84c7c9-bhh7z"]
Oct 31 13:00:12.424132 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.424096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.447115 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.437238    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 13:00:12.449988 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.449887    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cd84c7c9-bhh7z"]
Oct 31 13:00:12.464508 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.464481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.464615 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.464527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.464739 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.464714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.464774 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.464759    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.464858 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.464793    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.464858 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.464819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mc7\" (UniqueName: \"kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.464923 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.464880    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.464923 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.464906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.530059 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.530027    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t" event={"ID":"41aeee13-d6f6-4b22-b0d0-5c6a88038716","Type":"ContainerStarted","Data":"7c1d4b8931b94666de3faeb88062c0338b6e4dcf70d31359d5ba60fd97912843"}
Oct 31 13:00:12.532789 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.532654    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv" event={"ID":"2bcc9522-04bc-46ec-873c-ac4d00b96ffb","Type":"ContainerStarted","Data":"1b82593468fcc2aa1cde6c223a6d90b941c139cc32c8d0d91287b316ede3ef1f"}
Oct 31 13:00:12.532789 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.532751    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv" event={"ID":"2bcc9522-04bc-46ec-873c-ac4d00b96ffb","Type":"ContainerStarted","Data":"0b6ebd5d08f62cf17c60303fd4d9b82ccaddfad193603b82d849ae31f6114ead"}
Oct 31 13:00:12.536837 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.534275    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr" event={"ID":"be0506ab-10a8-4158-b7e1-62ded5ce9f11","Type":"ContainerStarted","Data":"f7eb57b0fd124d00b8fd47f901e74808fc8c745743269c37aa8fc4dc38cc0084"}
Oct 31 13:00:12.566087 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.566087 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.566307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.566307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566136    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mc7\" (UniqueName: \"kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.566307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.566307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.566307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.566307 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.567332 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.567332 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566917    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.567332 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.566960    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.567558 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.567360    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.567558 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.567455    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.569818 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.569755    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.571399 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.571378    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.580614 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.580575    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mc7\" (UniqueName: \"kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.758071 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.758031    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:12.963897 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.962882    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:12.963897 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.963749    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:12.971490 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:12.971255    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:13.049266 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:13.049192    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64cd84c7c9-bhh7z"]
Oct 31 13:00:13.055327 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:13.055304    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb015cd_93dd_4a5f_a8f2_c1192a389a81.slice/crio-c152106cf4fbd37551e2bdd4d53aad571ec529a904d046202a3c593c49c4b2e3 WatchSource:0}: Error finding container c152106cf4fbd37551e2bdd4d53aad571ec529a904d046202a3c593c49c4b2e3: Status 404 returned error can't find the container with id c152106cf4fbd37551e2bdd4d53aad571ec529a904d046202a3c593c49c4b2e3
Oct 31 13:00:13.542028 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:13.541996    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cd84c7c9-bhh7z" event={"ID":"1eb015cd-93dd-4a5f-a8f2-c1192a389a81","Type":"ContainerStarted","Data":"7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a"}
Oct 31 13:00:13.542200 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:13.542176    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cd84c7c9-bhh7z" event={"ID":"1eb015cd-93dd-4a5f-a8f2-c1192a389a81","Type":"ContainerStarted","Data":"c152106cf4fbd37551e2bdd4d53aad571ec529a904d046202a3c593c49c4b2e3"}
Oct 31 13:00:13.548347 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:13.548317    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv" event={"ID":"2bcc9522-04bc-46ec-873c-ac4d00b96ffb","Type":"ContainerStarted","Data":"6cb3383f1f2b236aedc5ee875d9d9d3d1a5c4007f24e42db9c55790e5ba64651"}
Oct 31 13:00:13.553080 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:13.553060    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:13.573411 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:13.572778    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-64cd84c7c9-bhh7z" podStartSLOduration=1.572757382 podStartE2EDuration="1.572757382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:13.571162127 +0000 UTC m=+64.968003511" watchObservedRunningTime="2026-10-31 13:00:13.572757382 +0000 UTC m=+64.969598762"
Oct 31 13:00:14.471065 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.470981    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:14.553692 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.553657    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr" event={"ID":"be0506ab-10a8-4158-b7e1-62ded5ce9f11","Type":"ContainerStarted","Data":"30d643f4dbb2b14a94ec7fa8e382c39afb9f3135e98671b067ff00e517be741b"}
Oct 31 13:00:14.553884 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.553740    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:14.558157 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.558127    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerStarted","Data":"18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92"}
Oct 31 13:00:14.558291 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.558187    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerStarted","Data":"470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40"}
Oct 31 13:00:14.558291 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.558221    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerStarted","Data":"7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13"}
Oct 31 13:00:14.558291 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.558244    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerStarted","Data":"8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034"}
Oct 31 13:00:14.558291 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.558268    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerStarted","Data":"862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20"}
Oct 31 13:00:14.558291 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.558291    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerStarted","Data":"88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f"}
Oct 31 13:00:14.559889 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.559859    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:14.560312 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.560289    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t" event={"ID":"41aeee13-d6f6-4b22-b0d0-5c6a88038716","Type":"ContainerStarted","Data":"563a255859e7de25a0461501b96c4622c4abe606fba11aa3feb5c55d391ad00b"}
Oct 31 13:00:14.563168 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.563145    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv" event={"ID":"2bcc9522-04bc-46ec-873c-ac4d00b96ffb","Type":"ContainerStarted","Data":"731e2e7e46fbbd352e0da486ba76603abb494e5b8a703a67f4ee1eeca6689cc6"}
Oct 31 13:00:14.563260 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.563212    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv" event={"ID":"2bcc9522-04bc-46ec-873c-ac4d00b96ffb","Type":"ContainerStarted","Data":"0a57981f1e700d6e46147a4c749b4d25918f8000b1572b96e5eb0bece8ecc48d"}
Oct 31 13:00:14.563260 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.563254    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv" event={"ID":"2bcc9522-04bc-46ec-873c-ac4d00b96ffb","Type":"ContainerStarted","Data":"11a4f34509204d6625c0357c409b5104d8b250a08704c05c057b40f728a0d13b"}
Oct 31 13:00:14.580369 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.580313    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr" podStartSLOduration=1.567749756 podStartE2EDuration="3.580295018s" totalImagesPullingTime="2.012545262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:14.579178276 +0000 UTC m=+65.976019658" watchObservedRunningTime="2026-10-31 13:00:14.580295018 +0000 UTC m=+65.977136401"
Oct 31 13:00:14.608834 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.608782    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv" podStartSLOduration=2.537609847 podStartE2EDuration="6.608766069s" totalImagesPullingTime="4.071156222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:14.607742598 +0000 UTC m=+66.004583978" watchObservedRunningTime="2026-10-31 13:00:14.608766069 +0000 UTC m=+66.005607453"
Oct 31 13:00:14.645496 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.645437    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.511651601 podStartE2EDuration="7.645422714s" totalImagesPullingTime="4.121849429s" totalInitContainerRuntime="11.921684ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:14.644955946 +0000 UTC m=+66.041797352" watchObservedRunningTime="2026-10-31 13:00:14.645422714 +0000 UTC m=+66.042264096"
Oct 31 13:00:14.702962 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.702904    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t" podStartSLOduration=1.439322127 podStartE2EDuration="3.702885945s" totalImagesPullingTime="2.263563818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:14.672022652 +0000 UTC m=+66.068864037" watchObservedRunningTime="2026-10-31 13:00:14.702885945 +0000 UTC m=+66.099727331"
Oct 31 13:00:14.789138 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.789050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:14.789284 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.789142    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 13:00:14.792491 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.792467    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 13:00:14.792491 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.792467    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 13:00:14.801973 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.801945    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:14.801973 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.801968    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 13:00:14.811066 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.811047    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lsjfn\""
Oct 31 13:00:14.818160 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.818136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 13:00:14.990968 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.990932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 13:00:14.994195 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:14.994171    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 13:00:15.004324 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.004305    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lt4jt\""
Oct 31 13:00:15.004415 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.004368    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 13:00:15.011511 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.011492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:15.014037 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.014008    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 13:00:15.101905 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.101170    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jbb9m"]
Oct 31 13:00:15.104447 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.103837    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g22bl\""
Oct 31 13:00:15.110683 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.110660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 13:00:15.287633 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.287604    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-c6hpz"]
Oct 31 13:00:15.290360 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:15.290337    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d66a46_02bb_4220_a78c_850e615072ca.slice/crio-aa6af958ee07b8fd1d408a030a96f37a1a2f0689e7439a384abd4927274cb66e WatchSource:0}: Error finding container aa6af958ee07b8fd1d408a030a96f37a1a2f0689e7439a384abd4927274cb66e: Status 404 returned error can't find the container with id aa6af958ee07b8fd1d408a030a96f37a1a2f0689e7439a384abd4927274cb66e
Oct 31 13:00:15.387994 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.387972    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-r4bj6"]
Oct 31 13:00:15.390480 ip-10-0-141-83 kubenswrapper[2576]: W1031 13:00:15.390453    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c98a803_2cc2_4806_8a14_b05c31ee7f03.slice/crio-a2cf5bf918e3d1cf231450dc91c8746348f456f10b643098022fc747cd52f353 WatchSource:0}: Error finding container a2cf5bf918e3d1cf231450dc91c8746348f456f10b643098022fc747cd52f353: Status 404 returned error can't find the container with id a2cf5bf918e3d1cf231450dc91c8746348f456f10b643098022fc747cd52f353
Oct 31 13:00:15.568225 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.568193    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-c6hpz" event={"ID":"97d66a46-02bb-4220-a78c-850e615072ca","Type":"ContainerStarted","Data":"aa6af958ee07b8fd1d408a030a96f37a1a2f0689e7439a384abd4927274cb66e"}
Oct 31 13:00:15.569457 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.569427    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbb9m" event={"ID":"e3f637a5-1cd2-41a6-98d6-ad45904012ad","Type":"ContainerStarted","Data":"894f6dcb43894df50f8e676515e89a41709249ec97001335c45892bc65c14f2d"}
Oct 31 13:00:15.570645 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.570614    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r4bj6" event={"ID":"3c98a803-2cc2-4806-8a14-b05c31ee7f03","Type":"ContainerStarted","Data":"a2cf5bf918e3d1cf231450dc91c8746348f456f10b643098022fc747cd52f353"}
Oct 31 13:00:15.571416 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:15.571397    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:16.577732 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:16.577639    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbb9m" event={"ID":"e3f637a5-1cd2-41a6-98d6-ad45904012ad","Type":"ContainerStarted","Data":"ddb0e3d7df9f596c8e1fd16ae024ff13ac7f1efcd27e417e345960421f3ec2ab"}
Oct 31 13:00:16.577732 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:16.577714    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbb9m" event={"ID":"e3f637a5-1cd2-41a6-98d6-ad45904012ad","Type":"ContainerStarted","Data":"e3f3546fab890bfb7539aab35a78545efb57a00804986edbe3003b29e3579e22"}
Oct 31 13:00:16.599936 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:16.599877    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jbb9m" podStartSLOduration=66.499186915 podStartE2EDuration="1m7.599857041s" totalImagesPullingTime="1.100670126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:16.597282069 +0000 UTC m=+67.994123446" watchObservedRunningTime="2026-10-31 13:00:16.599857041 +0000 UTC m=+67.996698425"
Oct 31 13:00:19.593775 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:19.593726    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-r4bj6" event={"ID":"3c98a803-2cc2-4806-8a14-b05c31ee7f03","Type":"ContainerStarted","Data":"3e751df9697521833e37f2461bcfe0b3b6f5c1ad624adce85f66ddaa4796cf85"}
Oct 31 13:00:19.594292 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:19.594249    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 13:00:19.614061 ip-10-0-141-83 kubenswrapper[2576]: I1031 13:00:19.614004    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-r4bj6" podStartSLOduration=67.189065802 podStartE2EDuration="1m10.613987133s" totalImagesPullingTime="3.424921331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 12:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:19.613329311 +0000 UTC m=+71.010170695" watchObservedRunningTime="2026-10-31 13:00:19.613987133 +0000 UTC m=+71.010828516"
Oct 31 13:00:20.513470 ip-10-0-141-83 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 13:00:20.528147 ip-10-0-141-83 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 13:00:20.528316 ip-10-0-141-83 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 13:00:20.528367 ip-10-0-141-83 systemd[1]: kubelet.service: Consumed 4.488s CPU time.
Oct 31 13:00:20.539400 ip-10-0-141-83 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 13:00:20.614136 ip-10-0-141-83 kubenswrapper[5787]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:00:20.614136 ip-10-0-141-83 kubenswrapper[5787]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 13:00:20.614136 ip-10-0-141-83 kubenswrapper[5787]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:00:20.614136 ip-10-0-141-83 kubenswrapper[5787]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 13:00:20.616253 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616160    5787 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:00:20.616253 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616252    5787 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:00:20.616253 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616256    5787 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616259    5787 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616262    5787 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616266    5787 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616269    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616273    5787 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616275    5787 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616278    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616281    5787 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616283    5787 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616286    5787 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616288    5787 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616291    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616293    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616296    5787 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616298    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616301    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616303    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616312    5787 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:00:20.616347 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616315    5787 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616318    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616321    5787 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616323    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616326    5787 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616328    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616331    5787 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616333    5787 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616336    5787 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616338    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616341    5787 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616344    5787 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616346    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616348    5787 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616351    5787 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616354    5787 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616357    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616359    5787 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616362    5787 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616365    5787 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:00:20.616866 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616368    5787 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616370    5787 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616372    5787 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616376    5787 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616381    5787 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616386    5787 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616389    5787 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616392    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616394    5787 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616397    5787 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616400    5787 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616402    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616405    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616407    5787 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616410    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616413    5787 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616416    5787 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616419    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616421    5787 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:00:20.617378 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616424    5787 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616427    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616430    5787 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616432    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616435    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616437    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616440    5787 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616442    5787 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616445    5787 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616447    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616450    5787 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616452    5787 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616455    5787 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616458    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616460    5787 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616463    5787 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616465    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616468    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616472    5787 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616474    5787 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:00:20.617878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616477    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616479    5787 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616481    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616484    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616486    5787 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616489    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616492    5787 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616495    5787 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616498    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616500    5787 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616503    5787 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616505    5787 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616508    5787 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616510    5787 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616513    5787 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616515    5787 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616518    5787 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616520    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616523    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616525    5787 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:00:20.618410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616529    5787 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616532    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616534    5787 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616536    5787 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616539    5787 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.616542    5787 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617350    5787 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617358    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617361    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617364    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617368    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617370    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617373    5787 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617375    5787 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617378    5787 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617381    5787 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617384    5787 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617386    5787 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617389    5787 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617392    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:00:20.618982 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617395    5787 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617398    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617400    5787 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617403    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617406    5787 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617408    5787 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617411    5787 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617415    5787 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617419    5787 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617423    5787 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617426    5787 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617429    5787 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617432    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617435    5787 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617438    5787 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617440    5787 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617443    5787 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617446    5787 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617449    5787 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:00:20.619493 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617452    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617455    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617457    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617460    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617462    5787 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617465    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617468    5787 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617470    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617473    5787 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617475    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617478    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617480    5787 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617483    5787 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617486    5787 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617489    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617491    5787 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617494    5787 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617496    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617498    5787 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617501    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:00:20.620015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617504    5787 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617506    5787 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617508    5787 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617514    5787 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617517    5787 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617519    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617522    5787 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617524    5787 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617527    5787 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617529    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617532    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617535    5787 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617538    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617541    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617544    5787 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617546    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617549    5787 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617551    5787 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617554    5787 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617556    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:00:20.620537 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617559    5787 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617561    5787 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617564    5787 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617566    5787 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617569    5787 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617572    5787 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617574    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617577    5787 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617580    5787 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617582    5787 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617585    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617587    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617590    5787 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617592    5787 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617594    5787 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617598    5787 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617601    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617603    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617605    5787 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617608    5787 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:00:20.621071 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617611    5787 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617615    5787 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617619    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617622    5787 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617624    5787 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617628    5787 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617631    5787 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617633    5787 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617636    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617642    5787 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617645    5787 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617647    5787 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.617650    5787 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618152    5787 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618157    5787 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618160    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618163    5787 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618166    5787 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618168    5787 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:00:20.621615 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618171    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618174    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618176    5787 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618178    5787 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618181    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618183    5787 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618186    5787 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618188    5787 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618192    5787 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618194    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618196    5787 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618199    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618201    5787 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618204    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618207    5787 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618209    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618212    5787 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618214    5787 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618217    5787 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618220    5787 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:00:20.622180 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618222    5787 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618225    5787 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618227    5787 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618229    5787 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618232    5787 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618234    5787 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618237    5787 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618239    5787 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618241    5787 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618244    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618247    5787 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618250    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618253    5787 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618257    5787 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618261    5787 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618263    5787 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618267    5787 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618270    5787 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618273    5787 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:00:20.623059 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618276    5787 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618279    5787 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618282    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618284    5787 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618287    5787 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618289    5787 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618292    5787 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618295    5787 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618298    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618301    5787 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618304    5787 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618306    5787 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618310    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618312    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618315    5787 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618317    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618320    5787 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618322    5787 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618325    5787 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618327    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:00:20.623928 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618329    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618332    5787 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618335    5787 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618338    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618340    5787 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618343    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618345    5787 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618347    5787 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618350    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618353    5787 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618355    5787 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618358    5787 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618360    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618362    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618365    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618367    5787 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618370    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618372    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618376    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:00:20.624877 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618378    5787 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618380    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618383    5787 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618385    5787 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618388    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618390    5787 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618393    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618396    5787 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618398    5787 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618400    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618403    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618405    5787 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618408    5787 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618411    5787 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618413    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618416    5787 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618418    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618421    5787 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618423    5787 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:00:20.625653 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618426    5787 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618428    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.618431    5787 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618524    5787 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618531    5787 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618537    5787 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618541    5787 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618545    5787 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618550    5787 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618555    5787 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618559    5787 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618563    5787 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618566    5787 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618569    5787 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618572    5787 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618575    5787 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618578    5787 flags.go:64] FLAG: --cgroup-root=""
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618581    5787 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618584    5787 flags.go:64] FLAG: --client-ca-file=""
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618587    5787 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618591    5787 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618616    5787 flags.go:64] FLAG: --cluster-domain=""
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618619    5787 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 13:00:20.626178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618623    5787 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618628    5787 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618631    5787 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618635    5787 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618638    5787 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618641    5787 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618645    5787 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618648    5787 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618651    5787 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618654    5787 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618657    5787 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618660    5787 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618664    5787 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618667    5787 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618670    5787 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618673    5787 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618676    5787 flags.go:64] FLAG: --enable-server="true"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618682    5787 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618685    5787 flags.go:64] FLAG: --event-burst="100"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618688    5787 flags.go:64] FLAG: --event-qps="50"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618691    5787 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618694    5787 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618698    5787 flags.go:64] FLAG: --eviction-hard=""
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618717    5787 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 13:00:20.626843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618720    5787 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618723    5787 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618726    5787 flags.go:64] FLAG: --eviction-soft=""
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618729    5787 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618732    5787 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618734    5787 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618738    5787 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618741    5787 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618744    5787 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618748    5787 flags.go:64] FLAG: --feature-gates=""
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618753    5787 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618756    5787 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618759    5787 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618763    5787 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618766    5787 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618769    5787 flags.go:64] FLAG: --help="false"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618771    5787 flags.go:64] FLAG: --hostname-override="ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618775    5787 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618778    5787 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618780    5787 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618784    5787 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618787    5787 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618790    5787 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 13:00:20.627442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618792    5787 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618795    5787 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618800    5787 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618803    5787 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618806    5787 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618808    5787 flags.go:64] FLAG: --kube-reserved=""
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618811    5787 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618814    5787 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618817    5787 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618820    5787 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618822    5787 flags.go:64] FLAG: --lock-file=""
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618825    5787 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618828    5787 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618831    5787 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618837    5787 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618839    5787 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618843    5787 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618846    5787 flags.go:64] FLAG: --logging-format="text"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618850    5787 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618854    5787 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618856    5787 flags.go:64] FLAG: --manifest-url=""
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618859    5787 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618864    5787 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618867    5787 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618870    5787 flags.go:64] FLAG: --max-pods="110"
Oct 31 13:00:20.628028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618873    5787 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618877    5787 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618879    5787 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618882    5787 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618885    5787 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618888    5787 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618891    5787 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618898    5787 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618902    5787 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618906    5787 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618909    5787 flags.go:64] FLAG: --pod-cidr=""
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618912    5787 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618915    5787 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618918    5787 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618920    5787 flags.go:64] FLAG: --port="10250"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618923    5787 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618926    5787 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0b4d28e464d8092f1"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618929    5787 flags.go:64] FLAG: --qos-reserved=""
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618932    5787 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618935    5787 flags.go:64] FLAG: --register-node="true"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618938    5787 flags.go:64] FLAG: --register-with-taints=""
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618941    5787 flags.go:64] FLAG: --registry-burst="10"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618944    5787 flags.go:64] FLAG: --registry-qps="5"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618947    5787 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618951    5787 flags.go:64] FLAG: --reserved-memory=""
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618959    5787 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 13:00:20.628679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618962    5787 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618964    5787 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618967    5787 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618970    5787 flags.go:64] FLAG: --runonce="false"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618973    5787 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618976    5787 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618979    5787 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618982    5787 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618985    5787 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618988    5787 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618991    5787 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618994    5787 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.618997    5787 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619000    5787 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619003    5787 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619007    5787 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619010    5787 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619012    5787 flags.go:64] FLAG: --system-cgroups=""
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619015    5787 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619020    5787 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619023    5787 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619026    5787 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619030    5787 flags.go:64] FLAG: --tls-min-version=""
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619032    5787 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619035    5787 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 13:00:20.629382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619038    5787 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619041    5787 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619043    5787 flags.go:64] FLAG: --v="2"
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619047    5787 flags.go:64] FLAG: --version="false"
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619052    5787 flags.go:64] FLAG: --vmodule=""
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619055    5787 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619060    5787 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619200    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619204    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619207    5787 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619210    5787 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619212    5787 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619215    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619217    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619220    5787 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619222    5787 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619225    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619227    5787 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619230    5787 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619232    5787 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619235    5787 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:00:20.630030 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619238    5787 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619241    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619245    5787 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619249    5787 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619252    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619254    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619257    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619259    5787 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619262    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619264    5787 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619267    5787 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619269    5787 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619272    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619274    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619277    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619279    5787 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619282    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619286    5787 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619289    5787 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:00:20.630551 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619292    5787 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619295    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619298    5787 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619301    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619304    5787 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619306    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619309    5787 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619311    5787 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619314    5787 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619316    5787 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619320    5787 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619323    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619327    5787 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619330    5787 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619334    5787 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619336    5787 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619339    5787 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619342    5787 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619344    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:00:20.631144 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619346    5787 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619349    5787 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619352    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619354    5787 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619356    5787 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619359    5787 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619361    5787 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619364    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619366    5787 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619369    5787 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619371    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619375    5787 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619377    5787 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619381    5787 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619384    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619386    5787 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619389    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619391    5787 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619393    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:00:20.631652 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619396    5787 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619398    5787 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619401    5787 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619403    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619406    5787 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619408    5787 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619411    5787 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619413    5787 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619417    5787 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619420    5787 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619422    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619425    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619427    5787 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619430    5787 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619432    5787 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619435    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619437    5787 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619440    5787 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619442    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619445    5787 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:00:20.632249 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619447    5787 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619449    5787 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619452    5787 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619454    5787 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619458    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619460    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619463    5787 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619466    5787 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619469    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619471    5787 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619474    5787 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619476    5787 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619478    5787 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619481    5787 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619483    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.619507    5787 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:00:20.632797 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619545    5787 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619549    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619552    5787 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619556    5787 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619558    5787 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619562    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619565    5787 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619568    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619570    5787 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619573    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619575    5787 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619578    5787 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619580    5787 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619583    5787 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619586    5787 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619590    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619594    5787 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619596    5787 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619599    5787 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:00:20.633189 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619602    5787 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619606    5787 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619608    5787 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619611    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619613    5787 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619616    5787 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619619    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619621    5787 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619624    5787 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619626    5787 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619628    5787 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619631    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619633    5787 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619636    5787 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619639    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619641    5787 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619644    5787 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619646    5787 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619650    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:00:20.633659 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619653    5787 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619655    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619658    5787 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619660    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619663    5787 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619666    5787 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619668    5787 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619671    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619673    5787 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619676    5787 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619678    5787 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619681    5787 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619683    5787 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619685    5787 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619689    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619692    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619694    5787 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619697    5787 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619712    5787 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619715    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:00:20.634298 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619718    5787 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619721    5787 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619723    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619725    5787 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619728    5787 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619731    5787 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619733    5787 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619736    5787 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619738    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619741    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619744    5787 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619748    5787 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619750    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619753    5787 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619755    5787 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619758    5787 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619760    5787 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619763    5787 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619765    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619768    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:00:20.634876 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619770    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619773    5787 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619775    5787 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619778    5787 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619780    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619783    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619786    5787 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619789    5787 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619791    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619793    5787 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619796    5787 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619798    5787 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619801    5787 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619804    5787 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619806    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619812    5787 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619815    5787 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619818    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619820    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:00:20.635374 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619823    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619825    5787 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619827    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619830    5787 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619833    5787 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619836    5787 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619838    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619841    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.619843    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.626223    5787 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.626240    5787 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.626265    5787 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.626270    5787 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626330    5787 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626335    5787 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626339    5787 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626342    5787 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626344    5787 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626347    5787 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:00:20.636052 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626349    5787 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626353    5787 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626355    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626358    5787 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626361    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626363    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626366    5787 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626369    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626371    5787 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626374    5787 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626376    5787 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626378    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626381    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626384    5787 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626386    5787 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626389    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626392    5787 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626395    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626397    5787 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:00:20.636535 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626400    5787 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626402    5787 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626406    5787 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626409    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626412    5787 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626414    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626417    5787 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626419    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626422    5787 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626424    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626428    5787 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626432    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626435    5787 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626438    5787 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626441    5787 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626443    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626446    5787 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626448    5787 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626451    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:00:20.637056 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626453    5787 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626456    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626459    5787 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626462    5787 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626464    5787 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626467    5787 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626469    5787 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626472    5787 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626474    5787 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626477    5787 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626480    5787 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626482    5787 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626484    5787 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626487    5787 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626490    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626492    5787 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626495    5787 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626497    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626500    5787 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626502    5787 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:00:20.637553 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626505    5787 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626507    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626510    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626512    5787 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626515    5787 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626517    5787 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626520    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626522    5787 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626525    5787 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626527    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626530    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626532    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626535    5787 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626538    5787 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626540    5787 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626543    5787 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626546    5787 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626548    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626551    5787 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626553    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:00:20.638075 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626556    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626558    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626561    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626563    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626566    5787 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626568    5787 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626571    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626573    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626575    5787 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626578    5787 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626580    5787 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626583    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626586    5787 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626589    5787 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626593    5787 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626596    5787 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626599    5787 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626601    5787 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626604    5787 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:00:20.638561 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626607    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626609    5787 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626612    5787 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.626641    5787 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626812    5787 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626817    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626820    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626822    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626825    5787 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626833    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626836    5787 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626839    5787 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626841    5787 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626844    5787 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626846    5787 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626849    5787 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:00:20.639051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626852    5787 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626855    5787 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626857    5787 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626860    5787 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626863    5787 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626866    5787 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626868    5787 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626870    5787 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626873    5787 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626876    5787 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626878    5787 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626881    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626884    5787 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626886    5787 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626888    5787 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626891    5787 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626894    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626897    5787 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626899    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:00:20.639451 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626901    5787 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626904    5787 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626906    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626908    5787 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626911    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626914    5787 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626916    5787 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626920    5787 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626924    5787 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626928    5787 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626930    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626933    5787 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626936    5787 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626938    5787 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626941    5787 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626943    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626945    5787 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626948    5787 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626950    5787 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:00:20.640016 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626952    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626957    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626960    5787 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626962    5787 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626964    5787 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626967    5787 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626970    5787 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626972    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626975    5787 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626977    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626980    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626983    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626986    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626988    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626991    5787 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626993    5787 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626995    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.626998    5787 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627001    5787 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627003    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:00:20.640546 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627005    5787 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627008    5787 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627011    5787 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627014    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627016    5787 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627019    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627021    5787 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627024    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627026    5787 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627029    5787 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627031    5787 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627035    5787 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627038    5787 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627043    5787 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627046    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627049    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627051    5787 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627054    5787 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627056    5787 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627059    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:00:20.641215 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627061    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627064    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627066    5787 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627069    5787 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627071    5787 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627074    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627076    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627079    5787 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627081    5787 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627083    5787 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627086    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627088    5787 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627091    5787 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627093    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627097    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627099    5787 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:00:20.641815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.627122    5787 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627154    5787 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627159    5787 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627162    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627165    5787 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627168    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627170    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627173    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627176    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627179    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627182    5787 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627185    5787 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627187    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627190    5787 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627192    5787 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627194    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627197    5787 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627200    5787 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627203    5787 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627205    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 13:00:20.642351 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627208    5787 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627210    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627213    5787 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627215    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627218    5787 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627220    5787 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627223    5787 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627227    5787 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627230    5787 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627233    5787 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627236    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627240    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627242    5787 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627245    5787 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627248    5787 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627250    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627252    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627255    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627257    5787 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627260    5787 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 13:00:20.643148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627262    5787 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627265    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627268    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627271    5787 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627273    5787 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627276    5787 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627278    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627280    5787 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627283    5787 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627286    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627288    5787 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627291    5787 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627293    5787 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627296    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627298    5787 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627301    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627303    5787 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627306    5787 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627308    5787 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 13:00:20.643814 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627310    5787 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627313    5787 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627315    5787 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627318    5787 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627320    5787 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627324    5787 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627327    5787 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627329    5787 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627332    5787 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627334    5787 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627336    5787 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627340    5787 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627342    5787 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627345    5787 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627347    5787 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627351    5787 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627354    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627357    5787 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627359    5787 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627363    5787 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 13:00:20.644301 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627366    5787 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627368    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627370    5787 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627373    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627375    5787 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627378    5787 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627380    5787 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627383    5787 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627386    5787 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627388    5787 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627390    5787 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627394    5787 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627398    5787 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627401    5787 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627403    5787 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627406    5787 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627408    5787 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627411    5787 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627413    5787 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 13:00:20.644975 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627416    5787 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627418    5787 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627421    5787 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627423    5787 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627426    5787 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627428    5787 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627430    5787 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627433    5787 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:20.627436    5787 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.627499    5787 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.628481    5787 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.628534    5787 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.628734    5787 server.go:1014] "Starting client certificate rotation"
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.628860    5787 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.628908    5787 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:54:08 +0000 UTC" deadline="2028-06-14 08:52:58.283094004 +0000 UTC"
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.628944    5787 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14203h52m37.654152068s"
Oct 31 13:00:20.645472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.629177    5787 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:00:20.645912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.630292    5787 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 13:00:20.645912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.631403    5787 log.go:25] "Validated CRI v1 runtime API"
Oct 31 13:00:20.645912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.633412    5787 log.go:25] "Validated CRI v1 image API"
Oct 31 13:00:20.645912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.634199    5787 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 13:00:20.645912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.636582    5787 fs.go:135] Filesystem UUIDs: map[40fc6b03-0e55-4023-a6c4-c1a60c76cd84:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cffb27ce-fbee-4270-b589-ff6bfc791d69:/dev/nvme0n1p3]
Oct 31 13:00:20.646065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.636598    5787 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0086b7a63de0798c31fadaf9c57d093beab2e50bbe256195f6cf1552a95c9f59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0086b7a63de0798c31fadaf9c57d093beab2e50bbe256195f6cf1552a95c9f59/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01cb702c3b971ea1c290687780c60bd36906a505086d0c552910122bdaf19955/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01cb702c3b971ea1c290687780c60bd36906a505086d0c552910122bdaf19955/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0736cae6a049b56e1016fcb9045064f8a3cb4b2cfd3ec2fb4c7973a886d7627a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0736cae6a049b56e1016fcb9045064f8a3cb4b2cfd3ec2fb4c7973a886d7627a/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b45337793c79920fff6e52e54fb47fb4d48eb004fa6b5c94a75d0410e99a507/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b45337793c79920fff6e52e54fb47fb4d48eb004fa6b5c94a75d0410e99a507/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1c114446240ba720521a0ff11a76b834dd50c44d2f2dce13ccdc332332939955/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1c114446240ba720521a0ff11a76b834dd50c44d2f2dce13ccdc332332939955/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24b898120cb835f08c3532e4b6190751f2455f3a41a6d3d9c1d4c43561ba438f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24b898120cb835f08c3532e4b6190751f2455f3a41a6d3d9c1d4c43561ba438f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/460569fc828b21f2b6acdb11f345c732ec3f9572f34eaa5f26c3d38e05778327/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/460569fc828b21f2b6acdb11f345c732ec3f9572f34eaa5f26c3d38e05778327/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63a978d5f32e285b86837234e75a0ceb01f2c6fda51fd8813ca0e2b7aa3db2e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63a978d5f32e285b86837234e75a0ceb01f2c6fda51fd8813ca0e2b7aa3db2e5/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72518e06fb622b5bdf461b87e80e3db32ad81a20fd610a0acfd2ea58fbebfc55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72518e06fb622b5bdf461b87e80e3db32ad81a20fd610a0acfd2ea58fbebfc55/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b77996449b59248e4f4dcbe8b045bc945865682ed50288dd5938055bb184253/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b77996449b59248e4f4dcbe8b045bc945865682ed50288dd5938055bb184253/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7c1d4b8931b94666de3faeb88062c0338b6e4dcf70d31359d5ba60fd97912843/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7c1d4b8931b94666de3faeb88062c0338b6e4dcf70d31359d5ba60fd97912843/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/804915a6fb388953ad84b69aa7fae1db20f2c525906b4282360e1e5be9df008d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/804915a6fb388953ad84b69aa7fae1db20f2c525906b4282360e1e5be9df008d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/87b7996c7cfc5cc6f7d60ae9b4cc8f0d0e44a0020897de5a8ee0585bc86a20c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/87b7996c7cfc5cc6f7d60ae9b4cc8f0d0e44a0020897de5a8ee0585bc86a20c5/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/894f6dcb43894df50f8e676515e89a41709249ec97001335c45892bc65c14f2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/894f6dcb43894df50f8e676515e89a41709249ec97001335c45892bc65c14f2d/userdata/shm major:0 minor:472 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d67c669e0d954f3b9e96cecfadc7a89b4aa3c07d161493e489c50661e2819f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d67c669e0d954f3b9e96cecfadc7a89b4aa3c07d161493e489c50661e2819f8/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/95d115eee69c9a5b55ffee64a98d09b67db14750517756dcb358958a57836c55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/95d115eee69c9a5b55ffee64a98d09b67db14750517756dcb358958a57836c55/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9dfe419f153032b25936bd01505811562dc0b0d58b0be0e406f90a9fee7bb889/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9dfe419f153032b25936bd01505811562dc0b0d58b0be0e406f90a9fee7bb889/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2cf5bf918e3d1cf231450dc91c8746348f456f10b643098022fc747cd52f353/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2cf5bf918e3d1cf231450dc91c8746348f456f10b643098022fc747cd52f353/userdata/shm major:0 minor:479 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa6af958ee07b8fd1d408a030a96f37a1a2f0689e7439a384abd4927274cb66e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa6af958ee07b8fd1d408a030a96f37a1a2f0689e7439a384abd4927274cb66e/userdata/shm major:0 minor:475 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1f5c29d61c9b98382520999638ed0794183b0998ffe6cdd3d6b9560947539b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1f5c29d61c9b98382520999638ed0794183b0998ffe6cdd3d6b9560947539b9/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b5d2d2ff275207a201d341e17482e44f97e6fe0cd09acce772a66e03331a8a79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b5d2d2ff275207a201d341e17482e44f97e6fe0cd09acce772a66e03331a8a79/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c152106cf4fbd37551e2bdd4d53aad571ec529a904d046202a3c593c49c4b2e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c152106cf4fbd37551e2bdd4d53aad571ec529a904d046202a3c593c49c4b2e3/userdata/shm major:0 minor:402 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1927855abaceab46e328d6f0151d935a379dbc4633a1cfb9704de5d825d242d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1927855abaceab46e328d6f0151d935a379dbc4633a1cfb9704de5d825d242d/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e403741246066bdfc00ee3aba1ca9f28c33660c88723714e5a8c4b68be50de39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e403741246066bdfc00ee3aba1ca9f28c33660c88723714e5a8c4b68be50de39/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee22a4e32efe46d0530853935902db4730f0f67b9f487ffefa275ec02aa40665/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee22a4e32efe46d0530853935902db4730f0f67b9f487ffefa275ec02aa40665/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7eb57b0fd124d00b8fd47f901e74808fc8c745743269c37aa8fc4dc38cc0084/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7eb57b0fd124d00b8fd47f901e74808fc8c745743269c37aa8fc4dc38cc0084/userdata/shm major:0 minor:379 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e7e4656-607a-4995-a250-a3bda30be0c1/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1e7e4656-607a-4995-a250-a3bda30be0c1/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e7e4656-607a-4995-a250-a3bda30be0c1/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1e7e4656-607a-4995-a250-a3bda30be0c1/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e7e4656-607a-4995-a250-a3bda30be0c1/volumes/kubernetes.io~projected/kube-api-access-9zkqv:{mountpoint:/var/lib/kubelet/pods/1e7e4656-607a-4995-a250-a3bda30be0c1/volumes/kubernetes.io~projected/kube-api-access-9zkqv major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes/kubernetes.io~projected/kube-api-access-p7mc7:{mountpoint:/var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes/kubernetes.io~projected/kube-api-access-p7mc7 major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:400 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/web-config/alertmanager/9 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/web-config/config-reloader/6 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~empty-dir/config-out major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~projected/kube-api-access-fx59n:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~projected/kube-api-access-fx59n major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~projected/tls-assets major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/config-volume major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/web-config major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~projected/kube-api-access-48b5c:{mountpoint:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~projected/kube-api-access-48b5c major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3814f51b-f30b-4f73-8061-ece6249a6edb/volumes/kubernetes.io~projected/kube-api-access-s2n6k:{mountpoint:/var/lib/kubelet/pods/3814f51b-f30b-4f73-8061-ece6249a6edb/volumes/kubernetes.io~projected/kube-api-access-s2n6k major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3814f51b-f30b-4f73-8061-ece6249a6edb/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3814f51b-f30b-4f73-8061-ece6249a6edb/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3814f51b-f30b-4f73-8061-ece6249a6edb/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/3814f51b-f30b-4f73-8061-ece6249a6edb/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3c98a803-2cc2-4806-8a14-b05c31ee7f03/volumes/kubernetes.io~projected/kube-api-access-t6db7:{mountpoint:/var/lib/kubelet/pods/3c98a803-2cc2-4806-8a14-b05c31ee7f03/volumes/kubernetes.io~projected/kube-api-access-t6db7 major:0 minor:474 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3dc748e5-08b1-4ffa-900c-09730777cc8d/volumes/kubernetes.io~projected/kube-api-access-xjtw4:{mountpoint:/var/lib/kubelet/pods/3dc748e5-08b1-4ffa-900c-09730777cc8d/volumes/kubernetes.io~projected/kube-api-access-xjtw4 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3dc748e5-08b1-4ffa-900c-09730777cc8d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/3dc748e5-08b1-4ffa-900c-09730777cc8d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~projected/kube-api-access-n2nhc:{mountpoint:/var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~projected/kube-api-access-n2nhc major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/448c900f-a9c5-4df1-b49e-e76269934da4/volumes/kubernetes.io~projected/kube-api-access-55jgb:{mountpoint:/var/lib/kubelet/pods/448c900f-a9c5-4df1-b49e-e76269934da4/volumes/kubernetes.io~projected/kube-api-access-55jgb major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5337ff71-42fc-4f7d-a92f-67e13f9c0a00/volumes/kubernetes.io~projected/kube-api-access-4gztt:{mountpoint:/var/lib/kubelet/pods/5337ff71-42fc-4f7d-a92f-67e13f9c0a00/volumes/kubernetes.io~projected/kube-api-access-4gztt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5510e6da-dd62-4c2f-b73e-75ca74cad962/volumes/kubernetes.io~projected/kube-api-access-lgzj5:{mountpoint:/var/lib/kubelet/pods/5510e6da-dd62-4c2f-b73e-75ca74cad962/volumes/kubernetes.io~projected/kube-api-access-lgzj5 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57294d98-42b2-47f0-a79a-d9cf7053ff9c/volumes/kubernetes.io~projected/kube-api-access-zh7k9:{mountpoint:/var/lib/kubelet/pods/57294d98-42b2-47f0-a79a-d9cf7053ff9c/volumes/kubernetes.io~projected/kube-api-access-zh7k9 major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57294d98-42b2-47f0-a79a-d9cf7053ff9c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/57294d98-42b2-47f0-a79a-d9cf7053ff9c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/57294d98-42b2-47f0-a79a-d9cf7053ff9c/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/57294d98-42b2-47f0-a79a-d9cf7053ff9c/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b16b0cb-c872-4fee-8a2a-83adb2777cc6/volumes/kubernetes.io~projected/kube-api-access-4h8pj:{mountpoint:/var/lib/kubelet/pods/5b16b0cb-c872-4fee-8a2a-83adb2777cc6/volumes/kubernetes.io~projected/kube-api-access-4h8pj major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c4a59dc-f603-4d34-a82a-23813a1b86b6/volumes/kubernetes.io~projected/kube-api-access-2ts4m:{mountpoint:/var/lib/kubelet/pods/5c4a59dc-f603-4d34-a82a-23813a1b86b6/volumes/kubernetes.io~projected/kube-api-access-2ts4m major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/686b38d0-13b9-4d2c-a1cf-5b8053804350/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/686b38d0-13b9-4d2c-a1cf-5b8053804350/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f24485c-3402-4cef-84eb-6ea5dd63be64/volumes/kubernetes.io~projected/kube-api-access-2frp7:{mountpoint:/var/lib/kubelet/pods/7f24485c-3402-4cef-84eb-6ea5dd63be64/volumes/kubernetes.io~projected/kube-api-access-2frp7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f24485c-3402-4cef-84eb-6ea5dd63be64/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7f24485c-3402-4cef-84eb-6ea5dd63be64/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87328843-fa88-4d5d-9904-5a3036d5bb10/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/87328843-fa88-4d5d-9904-5a3036d5bb10/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87328843-fa88-4d5d-9904-5a3036d5bb10/volumes/kubernetes.io~projected/kube-api-access-ztx68:{mountpoint:/var/lib/kubelet/pods/87328843-fa88-4d5d-9904-5a3036d5bb10/volumes/kubernetes.io~projected/kube-api-access-ztx68 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87328843-fa88-4d5d-9904-5a3036d5bb10/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/87328843-fa88-4d5d-9904-5a3036d5bb10/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b3fbffa-ba33-46db-9bdf-fcdf578955f6/volumes/kubernetes.io~projected/kube-api-access-rsq8j:{mountpoint:/var/lib/kubelet/pods/8b3fbffa-ba33-46db-9bdf-fcdf578955f6/volumes/kubernetes.io~projected/kube-api-access-rsq8j major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8b3fbffa-ba33-46db-9bdf-fcdf578955f6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/8b3fbffa-ba33-46db-9bdf-fcdf578955f6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97d66a46-02bb-4220-a78c-850e615072ca/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/97d66a46-02bb-4220-a78c-850e615072ca/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a81cd320-aea5-4bbf-9a40-f83f814180b5/volumes/kubernetes.io~projected/kube-api-access-w8rxv:{mountpoint:/var/lib/kubelet/pods/a81cd320-aea5-4bbf-9a40-f83f814180b5/volumes/kubernetes.io~projected/kube-api-access-w8rxv major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a81cd320-aea5-4bbf-9a40-f83f814180b5/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a81cd320-aea5-4bbf-9a40-f83f814180b5/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a81cd320-aea5-4bbf-9a40-f83f814180b5/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/a81cd320-aea5-4bbf-9a40-f83f814180b5/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc47f7a4-649c-4529-a608-4479486a59d8/volumes/kubernetes.io~projected/kube-api-access-q4666:{mountpoint:/var/lib/kubelet/pods/bc47f7a4-649c-4529-a608-4479486a59d8/volumes/kubernetes.io~projected/kube-api-access-q4666 major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc47f7a4-649c-4529-a608-4479486a59d8/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bc47f7a4-649c-4529-a608-4479486a59d8/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc47f7a4-649c-4529-a608-4479486a59d8/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/bc47f7a4-649c-4529-a608-4479486a59d8/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be0506ab-10a8-4158-b7e1-62ded5ce9f11/volumes/kubernetes.io~projected/kube-api-access-9nwdg:{mountpoint:/var/lib/kubelet/pods/be0506ab-10a8-4158-b7e1-62ded5ce9f11/volumes/kubernetes.io~projected/kube-api-access-9nwdg major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be0506ab-10a8-4158-b7e1-62ded5ce9f11/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/be0506ab-10a8-4158-b7e1-62ded5ce9f11/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e22c3df7-5789-4439-ba22-3a7d2b30c3fa/volumes/kubernetes.io~projected/kube-api-access-w4wj9:{mountpoint:/var/lib/kubelet/pods/e22c3df7-5789-4439-ba22-3a7d2b30c3fa/volumes/kubernetes.io~projected/kube-api-access-w4wj9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3f637a5-1cd2-41a6-98d6-ad45904012ad/volumes/kubernetes.io~projected/kube-api-access-k6lbt:{mountpoint:/var/lib/kubelet/pods/e3f637a5-1cd2-41a6-98d6-ad45904012ad/volumes/kubernetes.io~projected/kube-api-access-k6lbt major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e3f637a5-1cd2-41a6-98d6-ad45904012ad/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/e3f637a5-1cd2-41a6-98d6-ad45904012ad/volumes/kubernetes.io~secret/metrics-certs major:0 minor:471 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes/kubernetes.io~projected/kube-api-access-qprzm:{mountpoint:/var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes/kubernetes.io~projected/kube-api-access-qprzm major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:258 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/36078489427ff51feae860cafbb79a5b2dc551c32d91cf966d14dd5958dcae2c/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/35812cd46e75cf8512fcc8a6680195547fe8dc04f42765a7041f9201eba009d7/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/c5ac9ea2f91de262015a3e937dbda73969de30dc458ae0c218ed2c0e32377f63/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/eb6152facc8184224341482ae0d7d5cb5e8a264658296caa2cfed1de8e9ff566/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/c469d6ab037ebe5eab332f34ab59ff649052fb3c0797efa85da08e1695f91e1e/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/9e8e2b3b7e3e0625279ded5912244df4bed3581d090f161deb495996227fae22/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/bffcef58fe9cc00d227f6fbb16fcfda308da1a9279cae5257501a55eef3227ce/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/246f56d06a88ac587463e805f5dc2bca31c9cdc3dcf297bb027f2d6f2c7bd452/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/526489ccd64c06bfe38faca942eadadb97bad3998b4b949665b8a6cf5e9bcbf3/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/45448f68ac609cdb8e7a3bf3300c2e2f0621507f5a8d4f348f3fb9b7033fe878/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/28bcb54547069792004b4efd294077ce0c2d228cf61b04c1654464a3054eb3e4/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/02168ec1d0add5ab08e515a2a37c7a34d1beff320be9fc2664386cae3c66e7b5/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/67c1b841fe261177b60df1cb9d685bea9f96d6674a73fa293de76f648ab01ede/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/377f0377233c93bc7be1f392b7873cdc1f28470a06a8ab7ba2b3f81edc3ec7ef/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c20cdc35f0f392af48dddae85919aa24d0b8f2b31f5d8760f17267a6a909f612/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/58616bfb00671e21278d0f3c9fc5a6f22c3bfc76e2bcc0a7f8a53e7d51e090cb/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/cf631c8bfc608d388a57321e612c931d78af4b386d1604d019fe7e5e9203debf/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/137529c882de85ad4f0ebabfd270b9e5e9d52b362600519a63e4a7f1726b81af/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/df0eb80a51cccc686b11494892191a6b11bac642e8bb113e3e832769a92f836c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ede7a1ef4694c1e5bfdc4bbc77a5d4a379baec8629560e7e8ab508ff3e1e63c8/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/ac62cf1fb9c54dd612b9777cf611f5542af630a7c552fcf5c62af08160314278/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/bd5c80c9dc8090bd9fbc9ff53bd9bece91c1c4b74a588ec466ea28f37aed28f8/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/cda632d31d3039ca1085a029e3ab3dd0b1b907c06e149b805f5572299b1f2cb4/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/877eca49edd13e36a7da52f01ad794ba92b037ebbcc8a962df50f441d0424193/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/5c27dc7612a6f12a95aeb6335c3289e01d8404f9b7b790c1b85f688e5a2a0c70/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/a3715fd43a1494fba7ccf669ba4832e198411148ec65e56d529a42e4755c85f9/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/ed8c6cc3bd78bcebca1231ed7ebca05e66e531b1ee62d93d7e4b3e3d043c3bea/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/b3625105758b927111750985552ac16bf69df26c019417897e55c6d3d68d1564/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/37a305cce90a07db1370c0c99aae29cac9f296a2291cbe0543429c867ccef25a/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-242:{mountpoint:/var/lib/containers/storage/overlay/3350d70e9b497d7097ec2f4b05a13cbb9ef0c0548bf1af46e2b9f197c1930557/merged major:0 minor:242 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/0df8769e0980b80bc4dbab33dcec38e654672c63b72aeb2a8b6ce92bf5dbad28/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/31c49e8df0e8e35ed22dbdf469a9edfb2dab39de51fd80ee6fee34d017fc9779/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/9d84f3ed56bc6d560382050cb200137ee449adf4d55742601fb28b7ce43907f1/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/22cd5e2faaab6d3b1e21fe6a504bb79333c16a59b48550c8124c644308cb09f2/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/7a613bf9a23c721fc3a86843cddfed2cd44d4e12d6e27156771265fbefb2338b/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/cce3a0161ce2aed073fc629da26d2638eef8c374e4e62f52bc0c116951ecf8c5/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3e6380dd09cefb6031fd9d7313c08b87980311f629b5dcb61456b1f377cf7fbd/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/d3dc9eec8e92ebf3dbb2c292d03a2fe51a70ec1e581d188819538785a08fafc1/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/366e1ead86ddc605a988898496798d60af1ff527270cbee5e040e4b04e68afba/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/707d860c28a672e2aaa34d60a20a35485718c9a58a493c7ec6b0d14970b4149d/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/7e6ef9358aec0ba9c3b6f153588cf00061e60465d40192a046a6c910f2909b7c/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/e6ed832303572dd50a728fd7bf0bd6207f6d5d0e924804df5dca6b3f3272fb39/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/718a077a094fee9bb6dd7cde459e4df3611afa2610cbedde48d9adc67e051d9e/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/65169d2708d626be1ba3260364d5e6b0c0c7dbde5e827f28f761b5d8f27ad522/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/8b20405b6fc8bd8d8ec7d6c9a709e3eb56669a0e5850d5a43d872046745caa7a/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/bfca9953e802f38d768480bb3ed5a964e3f587fc853b01e133cc03dcd14af19c/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/886d4fcf88c3eef9f9b075d5c824c28644e84bf893ccf04ad9935c8957abbda5/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/2c409637c2e792850a2756637cb533760206039f5f1ceedd8c83ced730fd76d7/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/f07e70c6f3b3aa1a3028ac6f4e8394ec356fc912879eec03c07c81e1cbcbee43/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/0d27eeca97eb62d084b2057a6c6ea3da2b755fa2e22803e3d8d3ea9d0ea90606/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/5bf401828a79753df6275aeb2fa18881d11f68e91a6d756eb812258ab53011f0/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/ec8f0dbe4d3b6cd00c920ec4e1858a7f6037d52d84eb7321b09edae259c8238b/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/d727c3fb16ba54e9ea36d7b1fb17c96c437283c8cc3c40a297538d04bb54806e/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/200476c1b1d1c646a4ef8b4cb45f0b0115ff5c44a94f76c2c15b0ee752b4dd64/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/e43df8cb6236268dc5ddbbd8b7d432bbb4a5e634c5293411b1d44675ea585b14/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/abcbe1acdc131a6387e062139f86ccab8456f0ff9580cef7b7051641b251bd54/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/bca27228c0e33002a1fb96efd8c289f5ff88bf9c5a813e77540e2b8b09b0e466/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/0778204febd49a94359d64b9234625e9621ca0892421dc348f8638071dcd6391/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/353f120bb10c4c415cab152ca31d39410582bd161bfb5dc4a8bd4b9fa94e3e3d/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/016bc1447c11526d4a9644e2c59254af80c2a44dbf4065f21a6815151c94f3da/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-435:{mountpoint:/var/lib/containers/storage/overlay/69989d9c82e84a980065034643183fae23b37ed10561b910ccd1ad87725d2b99/merged major:0 minor:435 fsType:overlay blockSize:0} overlay_0-437:{mountpoint:/var/lib/containers/storage/overlay/3a31d5c36964af697bbca2a9f2c4ca3f770c9c79720f99f910b410b7a067ca8c/merged major:0 minor:437 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/99f49ca6022d74deae2f4d02b74fded9fed95cf72b474d4fef32b6d269d9649e/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-447:{mountpoint:/var/lib/containers/storage/overlay/d6c641b054c9457d11abec5cd2aa7a0d068849ed2dae962fd4b65867594dd7ec/merged major:0 minor:447 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/38366ead66881bcd5fb013826de210cead8ddcf5a4a8e7b3c4b5227289b199e1/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-460:{mountpoint:/var/lib/containers/storage/overlay/e06d7d0d4ba51506d299d3e276b1772921b46f510bf1654acda65c9e5663a524/merged major:0 minor:460 fsType:overlay blockSize:0} overlay_0-477:{mountpoint:/var/lib/containers/storage/overlay/152234c2e5ff06de65e060a62a74178f00bf37255ce346ec2fabe673e8267918/merged major:0 minor:477 fsType:overlay blockSize:0} overlay_0-481:{mountpoint:/var/lib/containers/storage/overlay/8373585ad90187f896db86f7e267d14e0d94e9d516f1cc9d1ae79e8667ddf1d8/merged major:0 minor:481 fsType:overlay blockSize:0} overlay_0-483:{mountpoint:/var/lib/containers/storage/overlay/7d3d2fab5a51cbc27e2646df4a1ee1694f0e68ded577f92050b1842b682e2521/merged major:0 minor:483 fsType:overlay blockSize:0} overlay_0-485:{mountpoint:/var/lib/containers/storage/overlay/9b6e7c4148b2d814f68bd5ab3b6be810213ee23d427912e2f1921db78422e63a/merged major:0 minor:485 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/c6ca97beea31b8f9dcde2faec85847f330fd2fe899cdaf658b1a33e260bbb306/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/f06c4d5be9bec3ae95f9c6e4e3a1b8d02c53ad285697f009372dc422d1152700/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/d9f3b8f6d0e3a0093558a624919b90d89534abeded96d33791ff6bb27390a18d/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-498:{mountpoint:/var/lib/containers/storage/overlay/7958208e998330a31b9ed68add9ce9cc6f7b98c743a06aeee881ff16c8e4339c/merged major:0 minor:498 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f6d25fb54b0b831c960094ff33eb3e096bab909328ddaebcb80cf79e16303580/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-69:{mountpoint:/var/lib/containers/storage/overlay/40b2dd7f2a2ef2f104d539f465d9b2ceb2fc938e7983eaa74dc0bdcdb5598276/merged major:0 minor:69 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/d2aff928e919c2cbee92e5864eafda9d60464f1d57ff007dbc613eda48f61227/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/a42c3f595a1ed19bed5c8691a6704b5baefe31eb14421058a0b565f8e0976229/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/9ad4abc99b43d6203f3da4c591a6caeab66d87c1cd7830195a9de5b328a8bff6/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/b19d2c50a24db49154d263348dca4c19f1d8450f9d05ed286c135f7ed6fbf279/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/90dce58f10a97f3e9a31ea88df1e27680f65b870cbba127168bd6de2fdf617ea/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5c366df4d47678b1fbffe6feb4324e6e0d2043da80811901f6cdf874b63134c4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3e6fed9d93b372591237f064baaacf0a4c7849c6646c8d3aec0da8d8a14a7f37/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 13:00:20.646948 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.646462    5787 manager.go:217] Machine: {Timestamp:2026-10-31 13:00:20.64599775 +0000 UTC m=+0.061939582 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100642 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c97f6689dfe5d62b79f82166e09a SystemUUID:ec21c97f-6689-dfe5-d62b-79f82166e09a BootID:297b49ba-e85e-4562-9743-adbd2ab573d8 Filesystems:[{Device:/var/lib/kubelet/pods/8b3fbffa-ba33-46db-9bdf-fcdf578955f6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2cf5bf918e3d1cf231450dc91c8746348f456f10b643098022fc747cd52f353/userdata/shm DeviceMajor:0 DeviceMinor:479 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87328843-fa88-4d5d-9904-5a3036d5bb10/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5510e6da-dd62-4c2f-b73e-75ca74cad962/volumes/kubernetes.io~projected/kube-api-access-lgzj5 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-69 DeviceMajor:0 DeviceMinor:69 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8b3fbffa-ba33-46db-9bdf-fcdf578955f6/volumes/kubernetes.io~projected/kube-api-access-rsq8j DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~projected/kube-api-access-n2nhc DeviceMajor:0 DeviceMinor:372 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/686b38d0-13b9-4d2c-a1cf-5b8053804350/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3c98a803-2cc2-4806-8a14-b05c31ee7f03/volumes/kubernetes.io~projected/kube-api-access-t6db7 DeviceMajor:0 DeviceMinor:474 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-481 DeviceMajor:0 DeviceMinor:481 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc47f7a4-649c-4529-a608-4479486a59d8/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:278 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:316 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72518e06fb622b5bdf461b87e80e3db32ad81a20fd610a0acfd2ea58fbebfc55/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57294d98-42b2-47f0-a79a-d9cf7053ff9c/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:400 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes/kubernetes.io~projected/kube-api-access-p7mc7 DeviceMajor:0 DeviceMinor:401 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f24485c-3402-4cef-84eb-6ea5dd63be64/volumes/kubernetes.io~projected/kube-api-access-2frp7 DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d67c669e0d954f3b9e96cecfadc7a89b4aa3c07d161493e489c50661e2819f8/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:350 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e7e4656-607a-4995-a250-a3bda30be0c1/volumes/kubernetes.io~projected/kube-api-access-9zkqv DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc47f7a4-649c-4529-a608-4479486a59d8/volumes/kubernetes.io~projected/kube-api-access-q4666 DeviceMajor:0 DeviceMinor:280 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:314 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:318 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24b898120cb835f08c3532e4b6190751f2455f3a41a6d3d9c1d4c43561ba438f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1927855abaceab46e328d6f0151d935a379dbc4633a1cfb9704de5d825d242d/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3dc748e5-08b1-4ffa-900c-09730777cc8d/volumes/kubernetes.io~projected/kube-api-access-xjtw4 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a81cd320-aea5-4bbf-9a40-f83f814180b5/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01cb702c3b971ea1c290687780c60bd36906a505086d0c552910122bdaf19955/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-435 DeviceMajor:0 DeviceMinor:435 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes/kubernetes.io~projected/kube-api-access-qprzm DeviceMajor:0 DeviceMinor:259 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:319 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:353 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:355 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1e7e4656-607a-4995-a250-a3bda30be0c1/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b5d2d2ff275207a201d341e17482e44f97e6fe0cd09acce772a66e03331a8a79/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~projected/kube-api-access-48b5c DeviceMajor:0 DeviceMinor:356 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/be0506ab-10a8-4158-b7e1-62ded5ce9f11/volumes/kubernetes.io~projected/kube-api-access-9nwdg DeviceMajor:0 DeviceMinor:378 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:396 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee22a4e32efe46d0530853935902db4730f0f67b9f487ffefa275ec02aa40665/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1c114446240ba720521a0ff11a76b834dd50c44d2f2dce13ccdc332332939955/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:322 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:351 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:319 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/1e7e4656-607a-4995-a250-a3bda30be0c1/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e22c3df7-5789-4439-ba22-3a7d2b30c3fa/volumes/kubernetes.io~projected/kube-api-access-w4wj9 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b77996449b59248e4f4dcbe8b045bc945865682ed50288dd5938055bb184253/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/95d115eee69c9a5b55ffee64a98d09b67db14750517756dcb358958a57836c55/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:352 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/804915a6fb388953ad84b69aa7fae1db20f2c525906b4282360e1e5be9df008d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/57294d98-42b2-47f0-a79a-d9cf7053ff9c/volumes/kubernetes.io~projected/kube-api-access-zh7k9 DeviceMajor:0 DeviceMinor:288 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:370 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-242 DeviceMajor:0 DeviceMinor:242 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~projected/kube-api-access-fx59n DeviceMajor:0 DeviceMinor:321 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:371 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/87b7996c7cfc5cc6f7d60ae9b4cc8f0d0e44a0020897de5a8ee0585bc86a20c5/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:254 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3814f51b-f30b-4f73-8061-ece6249a6edb/volumes/kubernetes.io~projected/kube-api-access-s2n6k DeviceMajor:0 DeviceMinor:289 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-477 DeviceMajor:0 DeviceMinor:477 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5337ff71-42fc-4f7d-a92f-67e13f9c0a00/volumes/kubernetes.io~projected/kube-api-access-4gztt DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5c4a59dc-f603-4d34-a82a-23813a1b86b6/volumes/kubernetes.io~projected/kube-api-access-2ts4m DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e403741246066bdfc00ee3aba1ca9f28c33660c88723714e5a8c4b68be50de39/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:258 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:310 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7eb57b0fd124d00b8fd47f901e74808fc8c745743269c37aa8fc4dc38cc0084/userdata/shm DeviceMajor:0 DeviceMinor:379 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1f5c29d61c9b98382520999638ed0794183b0998ffe6cdd3d6b9560947539b9/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa6af958ee07b8fd1d408a030a96f37a1a2f0689e7439a384abd4927274cb66e/userdata/shm DeviceMajor:0 DeviceMinor:475 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3814f51b-f30b-4f73-8061-ece6249a6edb/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9dfe419f153032b25936bd01505811562dc0b0d58b0be0e406f90a9fee7bb889/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41aeee13-d6f6-4b22-b0d0-5c6a88038716/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:369 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87328843-fa88-4d5d-9904-5a3036d5bb10/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/57294d98-42b2-47f0-a79a-d9cf7053ff9c/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:277 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-460 DeviceMajor:0 DeviceMinor:460 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87328843-fa88-4d5d-9904-5a3036d5bb10/volumes/kubernetes.io~projected/kube-api-access-ztx68 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63a978d5f32e285b86837234e75a0ceb01f2c6fda51fd8813ca0e2b7aa3db2e5/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c152106cf4fbd37551e2bdd4d53aad571ec529a904d046202a3c593c49c4b2e3/userdata/shm DeviceMajor:0 DeviceMinor:402 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f24485c-3402-4cef-84eb-6ea5dd63be64/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2bcc9522-04bc-46ec-873c-ac4d00b96ffb/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:354 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7c1d4b8931b94666de3faeb88062c0338b6e4dcf70d31359d5ba60fd97912843/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/448c900f-a9c5-4df1-b49e-e76269934da4/volumes/kubernetes.io~projected/kube-api-access-55jgb DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a81cd320-aea5-4bbf-9a40-f83f814180b5/volumes/kubernetes.io~projected/kube-api-access-w8rxv DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0086b7a63de0798c31fadaf9c57d093beab2e50bbe256195f6cf1552a95c9f59/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a81cd320-aea5-4bbf-9a40-f83f814180b5/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:315 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:320 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-483 DeviceMajor:0 DeviceMinor:483 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-498 DeviceMajor:0 DeviceMinor:498 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/460569fc828b21f2b6acdb11f345c732ec3f9572f34eaa5f26c3d38e05778327/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5b16b0cb-c872-4fee-8a2a-83adb2777cc6/volumes/kubernetes.io~projected/kube-api-access-4h8pj DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e3f637a5-1cd2-41a6-98d6-ad45904012ad/volumes/kubernetes.io~projected/kube-api-access-k6lbt DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/be0506ab-10a8-4158-b7e1-62ded5ce9f11/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:375 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:317 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3dc748e5-08b1-4ffa-900c-09730777cc8d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-437 DeviceMajor:0 DeviceMinor:437 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-447 DeviceMajor:0 DeviceMinor:447 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97d66a46-02bb-4220-a78c-850e615072ca/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:466 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/894f6dcb43894df50f8e676515e89a41709249ec97001335c45892bc65c14f2d/userdata/shm DeviceMajor:0 DeviceMinor:472 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc47f7a4-649c-4529-a608-4479486a59d8/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:279 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0736cae6a049b56e1016fcb9045064f8a3cb4b2cfd3ec2fb4c7973a886d7627a/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-485 DeviceMajor:0 DeviceMinor:485 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b45337793c79920fff6e52e54fb47fb4d48eb004fa6b5c94a75d0410e99a507/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3814f51b-f30b-4f73-8061-ece6249a6edb/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e3f637a5-1cd2-41a6-98d6-ad45904012ad/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:471 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0086b7a63de0798 MacAddress:52:be:91:91:a5:a7 Speed:10000 Mtu:8901} {Name:01cb702c3b971ea MacAddress:52:f6:75:2b:d9:6f Speed:10000 Mtu:8901} {Name:0736cae6a049b56 MacAddress:ee:d0:55:5a:a6:3c Speed:10000 Mtu:8901} {Name:1b45337793c7992 MacAddress:1a:97:c9:64:ce:6b Speed:10000 Mtu:8901} {Name:1c114446240ba72 MacAddress:3e:43:28:7e:90:13 Speed:10000 Mtu:8901} {Name:63a978d5f32e285 MacAddress:86:f2:c4:06:ca:9c Speed:10000 Mtu:8901} {Name:72518e06fb622b5 MacAddress:62:16:f2:0a:20:dc Speed:10000 Mtu:8901} {Name:7b77996449b5924 MacAddress:4e:1e:6d:90:bd:0f Speed:10000 Mtu:8901} {Name:7c1d4b8931b9466 MacAddress:92:89:92:f8:a6:5e Speed:10000 Mtu:8901} {Name:894f6dcb43894df MacAddress:4e:de:0b:d9:07:26 Speed:10000 Mtu:8901} {Name:8d67c669e0d954f MacAddress:a6:38:90:19:c0:7a Speed:10000 Mtu:8901} {Name:a2cf5bf918e3d1c MacAddress:9e:f2:ef:54:86:01 Speed:10000 Mtu:8901} {Name:aa6af958ee07b8f MacAddress:46:c7:be:9f:99:58 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:89:49:05 Speed:0 Mtu:9001} {Name:br-int MacAddress:c6:83:56:02:d0:50 Speed:0 Mtu:8901} {Name:c152106cf4fbd37 MacAddress:4a:6c:57:85:b2:d3 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:89:49:05 Speed:0 Mtu:9001} {Name:f7eb57b0fd124d0 MacAddress:0a:db:e4:41:16:4a Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:02:53:6d:95:62:b8 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:76:18:8c:cf:62:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 13:00:20.647178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.646952    5787 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 13:00:20.647178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647001    5787 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 13:00:20.647363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647337    5787 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 13:00:20.647495 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647365    5787 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-83.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 13:00:20.647536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647497    5787 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 13:00:20.647536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647505    5787 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 13:00:20.647536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647515    5787 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:00:20.647536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647533    5787 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 13:00:20.647643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647635    5787 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 13:00:20.647869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647859    5787 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 13:00:20.647985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.647976    5787 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 13:00:20.648020 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.648014    5787 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 13:00:20.648048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.648028    5787 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 13:00:20.648048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.648038    5787 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 13:00:20.648048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.648047    5787 kubelet.go:405] "Adding apiserver pod source"
Oct 31 13:00:20.648152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.648056    5787 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 13:00:20.648929 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.648912    5787 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 13:00:20.648929 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.648925    5787 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:00:20.649415 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.649399    5787 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 13:00:20.652239 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652226    5787 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 13:00:20.652289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652240    5787 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 13:00:20.652346 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652337    5787 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 13:00:20.652538 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652526    5787 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 13:00:20.652578 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652567    5787 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 13:00:20.652689 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652679    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 13:00:20.652748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652694    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 13:00:20.652748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652725    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 13:00:20.652748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652733    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 13:00:20.652748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652739    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 13:00:20.652748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652745    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 13:00:20.652748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652750    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 13:00:20.652952 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652756    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 13:00:20.652952 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652763    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 13:00:20.652952 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652769    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 13:00:20.652952 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652778    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 13:00:20.652952 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652788    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 13:00:20.652952 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652822    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 13:00:20.652952 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.652833    5787 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 13:00:20.656717 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.656688    5787 server.go:1263] "Started kubelet"
Oct 31 13:00:20.657644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.656797    5787 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 13:00:20.657644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.657307    5787 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 13:00:20.657644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.657387    5787 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 13:00:20.657306 ip-10-0-141-83 systemd[1]: Started Kubernetes Kubelet.
Oct 31 13:00:20.658251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.657718    5787 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 13:00:20.660559 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.660481    5787 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 13:00:20.662228 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.662208    5787 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 13:00:20.662443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.662424    5787 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 13:00:20.662916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.662873    5787 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:54:10 +0000 UTC" deadline="2028-07-11 15:05:57.01593188 +0000 UTC"
Oct 31 13:00:20.663060 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.662916    5787 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14858h5m36.353019808s"
Oct 31 13:00:20.663060 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.662923    5787 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 13:00:20.663060 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.662936    5787 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 13:00:20.663060 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:20.663043    5787 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-83.us-east-2.compute.internal\" not found"
Oct 31 13:00:20.666399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.666378    5787 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 13:00:20.666654 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.666637    5787 factory.go:55] Registering systemd factory
Oct 31 13:00:20.666766 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.666755    5787 factory.go:223] Registration of the systemd container factory successfully
Oct 31 13:00:20.668075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.668060    5787 factory.go:153] Registering CRI-O factory
Oct 31 13:00:20.668075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.668076    5787 factory.go:223] Registration of the crio container factory successfully
Oct 31 13:00:20.668220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.668130    5787 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 13:00:20.668220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.668149    5787 factory.go:103] Registering Raw factory
Oct 31 13:00:20.668220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.668163    5787 manager.go:1199] Started watching for new ooms in manager
Oct 31 13:00:20.668516 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.668499    5787 manager.go:319] Starting recovery of all containers
Oct 31 13:00:20.669539 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669499    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets" seLinuxMountContext=""
Oct 31 13:00:20.669617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669545    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bcc9522-04bc-46ec-873c-ac4d00b96ffb" volumeName="kubernetes.io/empty-dir/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-tmp" seLinuxMountContext=""
Oct 31 13:00:20.669617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669563    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41aeee13-d6f6-4b22-b0d0-5c6a88038716" volumeName="kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 31 13:00:20.669617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669575    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5337ff71-42fc-4f7d-a92f-67e13f9c0a00" volumeName="kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-daemon-config" seLinuxMountContext=""
Oct 31 13:00:20.669617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669588    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be0506ab-10a8-4158-b7e1-62ded5ce9f11" volumeName="kubernetes.io/secret/be0506ab-10a8-4158-b7e1-62ded5ce9f11-monitoring-plugin-cert" seLinuxMountContext=""
Oct 31 13:00:20.669617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669600    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e505bf1f-b560-4226-9a6e-fad97121cbfe" volumeName="kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm" seLinuxMountContext=""
Oct 31 13:00:20.669617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669613    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bcc9522-04bc-46ec-873c-ac4d00b96ffb" volumeName="kubernetes.io/projected/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-kube-api-access-48b5c" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669626    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bcc9522-04bc-46ec-873c-ac4d00b96ffb" volumeName="kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669640    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bcc9522-04bc-46ec-873c-ac4d00b96ffb" volumeName="kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669651    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f24485c-3402-4cef-84eb-6ea5dd63be64" volumeName="kubernetes.io/projected/7f24485c-3402-4cef-84eb-6ea5dd63be64-kube-api-access-2frp7" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669664    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b3fbffa-ba33-46db-9bdf-fcdf578955f6" volumeName="kubernetes.io/configmap/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669676    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b3fbffa-ba33-46db-9bdf-fcdf578955f6" volumeName="kubernetes.io/projected/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-api-access-rsq8j" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669687    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be0506ab-10a8-4158-b7e1-62ded5ce9f11" volumeName="kubernetes.io/projected/be0506ab-10a8-4158-b7e1-62ded5ce9f11-kube-api-access-9nwdg" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669698    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e22c3df7-5789-4439-ba22-3a7d2b30c3fa" volumeName="kubernetes.io/configmap/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-serviceca" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669741    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669761    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3c98a803-2cc2-4806-8a14-b05c31ee7f03" volumeName="kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669775    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41aeee13-d6f6-4b22-b0d0-5c6a88038716" volumeName="kubernetes.io/empty-dir/41aeee13-d6f6-4b22-b0d0-5c6a88038716-audit-log" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669787    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41aeee13-d6f6-4b22-b0d0-5c6a88038716" volumeName="kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-client-ca-bundle" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669799    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="448c900f-a9c5-4df1-b49e-e76269934da4" volumeName="kubernetes.io/empty-dir/448c900f-a9c5-4df1-b49e-e76269934da4-tmp-dir" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669812    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87328843-fa88-4d5d-9904-5a3036d5bb10" volumeName="kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669824    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b3fbffa-ba33-46db-9bdf-fcdf578955f6" volumeName="kubernetes.io/secret/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669834    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e505bf1f-b560-4226-9a6e-fad97121cbfe" volumeName="kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669847    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669859    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669871    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5337ff71-42fc-4f7d-a92f-67e13f9c0a00" volumeName="kubernetes.io/projected/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-kube-api-access-4gztt" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669881    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87328843-fa88-4d5d-9904-5a3036d5bb10" volumeName="kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-env-overrides" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669891    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87328843-fa88-4d5d-9904-5a3036d5bb10" volumeName="kubernetes.io/secret/87328843-fa88-4d5d-9904-5a3036d5bb10-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669899    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="448c900f-a9c5-4df1-b49e-e76269934da4" volumeName="kubernetes.io/projected/448c900f-a9c5-4df1-b49e-e76269934da4-kube-api-access-55jgb" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669910    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" volumeName="kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669918    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" volumeName="kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669939    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669956    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 31 13:00:20.669950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669966    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bcc9522-04bc-46ec-873c-ac4d00b96ffb" volumeName="kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669973    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3814f51b-f30b-4f73-8061-ece6249a6edb" volumeName="kubernetes.io/empty-dir/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-textfile" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669981    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3814f51b-f30b-4f73-8061-ece6249a6edb" volumeName="kubernetes.io/projected/3814f51b-f30b-4f73-8061-ece6249a6edb-kube-api-access-s2n6k" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669988    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" volumeName="kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.669995    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670002    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b16b0cb-c872-4fee-8a2a-83adb2777cc6" volumeName="kubernetes.io/projected/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kube-api-access-4h8pj" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670010    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="686b38d0-13b9-4d2c-a1cf-5b8053804350" volumeName="kubernetes.io/secret/686b38d0-13b9-4d2c-a1cf-5b8053804350-agent-certs" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670017    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f24485c-3402-4cef-84eb-6ea5dd63be64" volumeName="kubernetes.io/secret/7f24485c-3402-4cef-84eb-6ea5dd63be64-cert" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670025    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87328843-fa88-4d5d-9904-5a3036d5bb10" volumeName="kubernetes.io/projected/87328843-fa88-4d5d-9904-5a3036d5bb10-kube-api-access-ztx68" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670032    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e505bf1f-b560-4226-9a6e-fad97121cbfe" volumeName="kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670039    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bcc9522-04bc-46ec-873c-ac4d00b96ffb" volumeName="kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670046    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bcc9522-04bc-46ec-873c-ac4d00b96ffb" volumeName="kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-grpc-tls" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670054    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dc748e5-08b1-4ffa-900c-09730777cc8d" volumeName="kubernetes.io/secret/3dc748e5-08b1-4ffa-900c-09730777cc8d-metrics-tls" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670062    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41aeee13-d6f6-4b22-b0d0-5c6a88038716" volumeName="kubernetes.io/projected/41aeee13-d6f6-4b22-b0d0-5c6a88038716-kube-api-access-n2nhc" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670072    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5510e6da-dd62-4c2f-b73e-75ca74cad962" volumeName="kubernetes.io/projected/5510e6da-dd62-4c2f-b73e-75ca74cad962-kube-api-access-lgzj5" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670081    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc47f7a4-649c-4529-a608-4479486a59d8" volumeName="kubernetes.io/configmap/bc47f7a4-649c-4529-a608-4479486a59d8-metrics-client-ca" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670088    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc47f7a4-649c-4529-a608-4479486a59d8" volumeName="kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670096    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" volumeName="kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670105    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670113    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670121    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3814f51b-f30b-4f73-8061-ece6249a6edb" volumeName="kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670130    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dc748e5-08b1-4ffa-900c-09730777cc8d" volumeName="kubernetes.io/configmap/3dc748e5-08b1-4ffa-900c-09730777cc8d-config-volume" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670138    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57294d98-42b2-47f0-a79a-d9cf7053ff9c" volumeName="kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670148    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81cd320-aea5-4bbf-9a40-f83f814180b5" volumeName="kubernetes.io/configmap/a81cd320-aea5-4bbf-9a40-f83f814180b5-metrics-client-ca" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670185    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3f637a5-1cd2-41a6-98d6-ad45904012ad" volumeName="kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670192    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e7e4656-607a-4995-a250-a3bda30be0c1" volumeName="kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-tuned" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670216    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670224    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670231    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5337ff71-42fc-4f7d-a92f-67e13f9c0a00" volumeName="kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cni-binary-copy" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670245    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97d66a46-02bb-4220-a78c-850e615072ca" volumeName="kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670252    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81cd320-aea5-4bbf-9a40-f83f814180b5" volumeName="kubernetes.io/projected/a81cd320-aea5-4bbf-9a40-f83f814180b5-kube-api-access-w8rxv" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670258    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81cd320-aea5-4bbf-9a40-f83f814180b5" volumeName="kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-tls" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670266    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc47f7a4-649c-4529-a608-4479486a59d8" volumeName="kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670274    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" volumeName="kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670281    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41aeee13-d6f6-4b22-b0d0-5c6a88038716" volumeName="kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670288    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57294d98-42b2-47f0-a79a-d9cf7053ff9c" volumeName="kubernetes.io/projected/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-api-access-zh7k9" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670296    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e505bf1f-b560-4226-9a6e-fad97121cbfe" volumeName="kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670304    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" volumeName="kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670313    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" volumeName="kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670320    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dc748e5-08b1-4ffa-900c-09730777cc8d" volumeName="kubernetes.io/empty-dir/3dc748e5-08b1-4ffa-900c-09730777cc8d-tmp-dir" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670328    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41aeee13-d6f6-4b22-b0d0-5c6a88038716" volumeName="kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670335    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57294d98-42b2-47f0-a79a-d9cf7053ff9c" volumeName="kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670350    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57294d98-42b2-47f0-a79a-d9cf7053ff9c" volumeName="kubernetes.io/empty-dir/57294d98-42b2-47f0-a79a-d9cf7053ff9c-volume-directive-shadow" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670357    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c4a59dc-f603-4d34-a82a-23813a1b86b6" volumeName="kubernetes.io/configmap/5c4a59dc-f603-4d34-a82a-23813a1b86b6-iptables-alerter-script" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670365    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c4a59dc-f603-4d34-a82a-23813a1b86b6" volumeName="kubernetes.io/projected/5c4a59dc-f603-4d34-a82a-23813a1b86b6-kube-api-access-2ts4m" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670372    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e7e4656-607a-4995-a250-a3bda30be0c1" volumeName="kubernetes.io/projected/1e7e4656-607a-4995-a250-a3bda30be0c1-kube-api-access-9zkqv" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670379    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" volumeName="kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670386    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5510e6da-dd62-4c2f-b73e-75ca74cad962" volumeName="kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-binary-copy" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670395    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="686b38d0-13b9-4d2c-a1cf-5b8053804350" volumeName="kubernetes.io/configmap/686b38d0-13b9-4d2c-a1cf-5b8053804350-konnectivity-ca" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670402    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87328843-fa88-4d5d-9904-5a3036d5bb10" volumeName="kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670410    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a81cd320-aea5-4bbf-9a40-f83f814180b5" volumeName="kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670417    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e505bf1f-b560-4226-9a6e-fad97121cbfe" volumeName="kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670429    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3814f51b-f30b-4f73-8061-ece6249a6edb" volumeName="kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-tls" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670438    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e7e4656-607a-4995-a250-a3bda30be0c1" volumeName="kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-tmp" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670446    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5510e6da-dd62-4c2f-b73e-75ca74cad962" volumeName="kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670452    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57294d98-42b2-47f0-a79a-d9cf7053ff9c" volumeName="kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-tls" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670461    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc47f7a4-649c-4529-a608-4479486a59d8" volumeName="kubernetes.io/projected/bc47f7a4-649c-4529-a608-4479486a59d8-kube-api-access-q4666" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670468    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e505bf1f-b560-4226-9a6e-fad97121cbfe" volumeName="kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670477    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670484    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bcc9522-04bc-46ec-873c-ac4d00b96ffb" volumeName="kubernetes.io/configmap/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-metrics-client-ca" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670492    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3814f51b-f30b-4f73-8061-ece6249a6edb" volumeName="kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-metrics-client-ca" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670525    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3814f51b-f30b-4f73-8061-ece6249a6edb" volumeName="kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670538    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8b3fbffa-ba33-46db-9bdf-fcdf578955f6" volumeName="kubernetes.io/empty-dir/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-data-volume" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670550    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e22c3df7-5789-4439-ba22-3a7d2b30c3fa" volumeName="kubernetes.io/projected/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-kube-api-access-w4wj9" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670561    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e505bf1f-b560-4226-9a6e-fad97121cbfe" volumeName="kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670570    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ab4d241-bf18-44f2-a803-1de48ef9086e" volumeName="kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670580    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bcc9522-04bc-46ec-873c-ac4d00b96ffb" volumeName="kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670589    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3dc748e5-08b1-4ffa-900c-09730777cc8d" volumeName="kubernetes.io/projected/3dc748e5-08b1-4ffa-900c-09730777cc8d-kube-api-access-xjtw4" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670598    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41aeee13-d6f6-4b22-b0d0-5c6a88038716" volumeName="kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-tls" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670605    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5510e6da-dd62-4c2f-b73e-75ca74cad962" volumeName="kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670613    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="57294d98-42b2-47f0-a79a-d9cf7053ff9c" volumeName="kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-metrics-client-ca" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670623    5787 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e3f637a5-1cd2-41a6-98d6-ad45904012ad" volumeName="kubernetes.io/projected/e3f637a5-1cd2-41a6-98d6-ad45904012ad-kube-api-access-k6lbt" seLinuxMountContext=""
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670633    5787 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 13:00:20.671244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.670638    5787 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 13:00:20.676065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.676041    5787 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:00:20.680988 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.680967    5787 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 13:00:20.681108 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.681024    5787 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:00:20.684648 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.684452    5787 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:00:20.695932 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.695903    5787 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 13:00:20.697992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.697973    5787 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 13:00:20.697992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.697991    5787 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 13:00:20.698177 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.698008    5787 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 13:00:20.698177 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:20.698036    5787 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 13:00:20.702542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.702517    5787 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 13:00:20.726576 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.726535    5787 generic.go:356] "Generic (PLEG): container finished" podID="3814f51b-f30b-4f73-8061-ece6249a6edb" containerID="b0fee4f73f846221ec445f4d95b125d00ae8cd357376aee766c102ad7a9769ff" exitCode=0
Oct 31 13:00:20.747699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.747660    5787 generic.go:356] "Generic (PLEG): container finished" podID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerID="b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055" exitCode=0
Oct 31 13:00:20.755014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.754979    5787 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="fd9b35c46c223a485ac1f35f2322bd37f78e4e3a48ff042b6f49ba9d00b635d6" exitCode=0
Oct 31 13:00:20.755014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.755009    5787 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="dd920d67b57f04fa7777d1bb8b6c2c4f003e1091571a5bc064b7b0f003306d0c" exitCode=0
Oct 31 13:00:20.755014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.755019    5787 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="62516776e5fc5ee27cc7fda29f0f5744e7f5eb5920e57fa55fe7f71914814d06" exitCode=0
Oct 31 13:00:20.755261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.755029    5787 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="3341e8d2059416a577e4c6f9c96068a94bc1f0e2afc0728b6726b26a3f129e7b" exitCode=0
Oct 31 13:00:20.755261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.755038    5787 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="b08d62ab1647aed24a17736fee085ea9a41a5b50da9faf49cc1b52f974d1392a" exitCode=0
Oct 31 13:00:20.755261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.755044    5787 generic.go:356] "Generic (PLEG): container finished" podID="5510e6da-dd62-4c2f-b73e-75ca74cad962" containerID="45583a5d6f0b23709f4d7d26e53fb46a2ba3e0b8cbdc83a50b2c4ca34013ba5e" exitCode=0
Oct 31 13:00:20.771698 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.771622    5787 generic.go:356] "Generic (PLEG): container finished" podID="3754ca7849e18e0d20b7fa7b829435b5" containerID="37bb83e7caa761b220e06ff2abea8f856dd681baff02af97deb6f50cc95735af" exitCode=0
Oct 31 13:00:20.791161 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.791134    5787 manager.go:324] Recovery completed
Oct 31 13:00:20.798126 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:20.798098    5787 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 13:00:20.815621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815593    5787 cpu_manager.go:226] "Starting" policy="none"
Oct 31 13:00:20.815621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815609    5787 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 13:00:20.815797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815633    5787 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 13:00:20.815848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815793    5787 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 13:00:20.815848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815810    5787 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 13:00:20.815848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815828    5787 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 13:00:20.815848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815834    5787 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 13:00:20.815848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815839    5787 policy_none.go:50] "Start"
Oct 31 13:00:20.815848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815848    5787 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 13:00:20.816002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.815865    5787 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 13:00:20.816926 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.816911    5787 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 13:00:20.816970 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.816928    5787 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 13:00:20.816970 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.816935    5787 policy_none.go:44] "Start"
Oct 31 13:00:20.824297 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.824261    5787 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 13:00:20.827365 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.827350    5787 manager.go:341] "Starting Device Plugin manager"
Oct 31 13:00:20.827863 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:20.827383    5787 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 13:00:20.827863 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.827863    5787 server.go:87] "Starting device plugin registration server"
Oct 31 13:00:20.828129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.828119    5787 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 13:00:20.828172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.828131    5787 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 13:00:20.828238 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.828212    5787 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 13:00:20.828374 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.828330    5787 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 13:00:20.828465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.828380    5787 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 13:00:20.828465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.828388    5787 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 13:00:20.828569 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.828449    5787 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T13:00:20.828367513Z","UUID":"4b89e348-44a2-4ace-811c-83949cfed2a9","Handler":null,"Name":"","Endpoint":""}
Oct 31 13:00:20.829868 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.829851    5787 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 13:00:20.829868 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.829872    5787 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 13:00:20.830266 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:20.830228    5787 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 13:00:20.930269 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.930234    5787 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 13:00:20.936980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.936931    5787 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 13:00:20.937135 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.937042    5787 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 13:00:20.937135 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.937062    5787 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-83.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 13:00:20.937230 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.937171    5787 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:20.951167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.951140    5787 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:20.951284 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.951214    5787 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:20.999024 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.998953    5787 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"]
Oct 31 13:00:20.999633 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.999583    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"7ebdbb4e6bd986d2f09f581a50c7d1f8","Type":"ContainerStarted","Data":"c9224fb1fce494c15fd6516e0acf6417d336adc8131437e8b08cb2a4cb8862d5"}
Oct 31 13:00:20.999778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.999651    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"7ebdbb4e6bd986d2f09f581a50c7d1f8","Type":"ContainerStarted","Data":"9dfe419f153032b25936bd01505811562dc0b0d58b0be0e406f90a9fee7bb889"}
Oct 31 13:00:20.999778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.999763    5787 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:20.999895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.999830    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"3754ca7849e18e0d20b7fa7b829435b5","Type":"ContainerStarted","Data":"241f8fe09639f15e97080acac5cd0be0a43235bb4de569439d579b81482c043e"}
Oct 31 13:00:20.999895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.999838    5787 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:20.999895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.999856    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"3754ca7849e18e0d20b7fa7b829435b5","Type":"ContainerDied","Data":"37bb83e7caa761b220e06ff2abea8f856dd681baff02af97deb6f50cc95735af"}
Oct 31 13:00:20.999895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:20.999876    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal" event={"ID":"3754ca7849e18e0d20b7fa7b829435b5","Type":"ContainerStarted","Data":"460569fc828b21f2b6acdb11f345c732ec3f9572f34eaa5f26c3d38e05778327"}
Oct 31 13:00:21.015243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.015218    5787 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:00:21.015343 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.015245    5787 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:00:21.015343 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:21.015272    5787 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.015343 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:21.015285    5787 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.082690 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.082605    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ebdbb4e6bd986d2f09f581a50c7d1f8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"7ebdbb4e6bd986d2f09f581a50c7d1f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.082690 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.082637    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.082690 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.082653    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.183301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.183263    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ebdbb4e6bd986d2f09f581a50c7d1f8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"7ebdbb4e6bd986d2f09f581a50c7d1f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.183301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.183302    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.183527 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.183329    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.183527 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.183374    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7ebdbb4e6bd986d2f09f581a50c7d1f8-config\") pod \"kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"7ebdbb4e6bd986d2f09f581a50c7d1f8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.183527 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.183379    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.183527 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.183380    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3754ca7849e18e0d20b7fa7b829435b5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" (UID: \"3754ca7849e18e0d20b7fa7b829435b5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.649141 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.649106    5787 apiserver.go:52] "Watching apiserver"
Oct 31 13:00:21.665316 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.665282    5787 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 13:00:21.666039 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.666014    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-64cd84c7c9-bhh7z","openshift-dns/dns-default-wwg8z","openshift-monitoring/metrics-server-b9f69b5fb-mnd5t","openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm","openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9","openshift-network-diagnostics/network-check-target-r4bj6","openshift-cluster-node-tuning-operator/tuned-5pwr7","openshift-ingress-canary/ingress-canary-2798k","openshift-insights/insights-runtime-extractor-nkxs6","openshift-dns/node-resolver-ld8vz","openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr","openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv","openshift-network-operator/iptables-alerter-4p8sf","openshift-monitoring/alertmanager-main-0","openshift-monitoring/node-exporter-srtvx","kube-system/global-pull-secret-syncer-c6hpz","openshift-console/console-f4dfdf7b8-7hlkd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb","openshift-ovn-kubernetes/ovnkube-node-cbknb","openshift-image-registry/node-ca-lvtzl","openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk","openshift-multus/multus-additional-cni-plugins-6rkkf","openshift-multus/network-metrics-daemon-jbb9m","kube-system/konnectivity-agent-phbg5","kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal","openshift-multus/multus-7pzvw"]
Oct 31 13:00:21.672617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.672582    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-lt4jt\""
Oct 31 13:00:21.672617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.672591    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.672857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.672660    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7lhn5\""
Oct 31 13:00:21.672857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.672689    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.672857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.672746    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-2x4wq\""
Oct 31 13:00:21.673482 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.673450    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 13:00:21.673666 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.673465    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.674045 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.674015    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 13:00:21.674144 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.674122    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-g22bl\""
Oct 31 13:00:21.674895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.674872    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 13:00:21.675358 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.675335    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 13:00:21.675671 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.675647    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.675775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.675674    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 13:00:21.675835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.675782    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 13:00:21.675884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.675859    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.675999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.675980    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 13:00:21.676069 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676047    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 13:00:21.676250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676231    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lsjfn\""
Oct 31 13:00:21.676307 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676255    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.676307 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676261    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 13:00:21.676465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676446    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.676587 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676570    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 13:00:21.676645 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676597    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mw6wr\""
Oct 31 13:00:21.676693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676643    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.676693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676651    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.676842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676824    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 13:00:21.676893 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676602    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-5b9kz\""
Oct 31 13:00:21.676940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676909    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.676940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676914    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.677043 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676990    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 13:00:21.677087 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.677056    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 13:00:21.677484 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.676047    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.677592 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.677543    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.677592 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.677560    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.677837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.677823    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.678318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678300    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vtxsh\""
Oct 31 13:00:21.678441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678319    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-drb8j\""
Oct 31 13:00:21.678441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678333    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 13:00:21.678441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678360    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 13:00:21.678441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678420    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 13:00:21.678749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678447    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.678749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678509    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-gth9v\""
Oct 31 13:00:21.678749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678532    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 13:00:21.678749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678581    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qpv6m\""
Oct 31 13:00:21.678749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678633    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.678749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678636    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ftfm5\""
Oct 31 13:00:21.678749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678742    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678774    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678809    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678860    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678878    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678932    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678962    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678970    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.678991    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-d9gnb\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.679045    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.679079    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 13:00:21.679119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.679111    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 13:00:21.679494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.679128    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2xlcn\""
Oct 31 13:00:21.679494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.679174    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hwtqn\""
Oct 31 13:00:21.679494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.679178    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2kvxk\""
Oct 31 13:00:21.679494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.679310    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 13:00:21.685535 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.685514    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-dbus\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:21.685629 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.685545    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:21.685629 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.685567    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c4a59dc-f603-4d34-a82a-23813a1b86b6-iptables-alerter-script\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 13:00:21.685629 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.685582    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c4a59dc-f603-4d34-a82a-23813a1b86b6-host-slash\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 13:00:21.685769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.685627    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ts4m\" (UniqueName: \"kubernetes.io/projected/5c4a59dc-f603-4d34-a82a-23813a1b86b6-kube-api-access-2ts4m\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 13:00:21.685769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.685689    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-kubelet-config\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:21.685908 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.685890    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5c4a59dc-f603-4d34-a82a-23813a1b86b6-iptables-alerter-script\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 13:00:21.685986 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.685969    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/97d66a46-02bb-4220-a78c-850e615072ca-original-pull-secret\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:21.699102 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.699071    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ts4m\" (UniqueName: \"kubernetes.io/projected/5c4a59dc-f603-4d34-a82a-23813a1b86b6-kube-api-access-2ts4m\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 13:00:21.769610 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.769578    5787 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 13:00:21.776141 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776104    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 13:00:21.776383 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776313    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 31 13:00:21.776451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776331    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 13:00:21.776525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776510    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l8r9q\""
Oct 31 13:00:21.776574 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776558    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 13:00:21.776574 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776566    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 13:00:21.776843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776820    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fm9ml\""
Oct 31 13:00:21.776843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776830    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 13:00:21.776982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776914    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 31 13:00:21.776982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776962    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 13:00:21.776982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776975    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 13:00:21.777085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.776975    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jfvtg\""
Oct 31 13:00:21.777085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.777025    5787 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.777159 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.777143    5787 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.777946 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.777930    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-esg1vsh0ffr3k\""
Oct 31 13:00:21.777946 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.777940    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 31 13:00:21.778037 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.777941    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 31 13:00:21.778655 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.778630    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 13:00:21.778853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.778838    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 13:00:21.779112 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.778857    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 13:00:21.779372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.779349    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 13:00:21.780124 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780091    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 31 13:00:21.780320 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780302    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 31 13:00:21.780390 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780352    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 31 13:00:21.780390 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780379    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780504    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lz75d\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780541    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780551    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780646    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780651    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780748    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780828    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.780851    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.781030    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.781061    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.781129    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.781186    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-q5jtb\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.781276    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.781277    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c65ps\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.781656    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-2fjdc\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.781752    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.781807    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.782289    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4r7h54n5eni9s\""
Oct 31 13:00:21.782938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.782443    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7zf52\""
Oct 31 13:00:21.786161 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786135    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.786252 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786177    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.786252 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786206    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.786252 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786228    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.786252 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786245    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2n6k\" (UniqueName: \"kubernetes.io/projected/3814f51b-f30b-4f73-8061-ece6249a6edb-kube-api-access-s2n6k\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.786439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786270    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-conf\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.786439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786294    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zkqv\" (UniqueName: \"kubernetes.io/projected/1e7e4656-607a-4995-a250-a3bda30be0c1-kube-api-access-9zkqv\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.786439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786317    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-ovn\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.786439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786342    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786444    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786465    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gztt\" (UniqueName: \"kubernetes.io/projected/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-kube-api-access-4gztt\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786485    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786504    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786537    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be0506ab-10a8-4158-b7e1-62ded5ce9f11-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786567    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysconfig\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786580    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786593    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-lib-modules\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786619    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-tuned\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.786632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786621    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786641    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-tmp\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786678    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-crio-socket\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786740    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-tmp\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786803    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-tuned\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786887    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-netns\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786937    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/be0506ab-10a8-4158-b7e1-62ded5ce9f11-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.786989    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-var-lib-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787034    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787069    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-kubernetes\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787093    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.787111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787093    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787132    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787164    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f24485c-3402-4cef-84eb-6ea5dd63be64-cert\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787240    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787274    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rxv\" (UniqueName: \"kubernetes.io/projected/a81cd320-aea5-4bbf-9a40-f83f814180b5-kube-api-access-w8rxv\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787286    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787302    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx59n\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787331    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/448c900f-a9c5-4df1-b49e-e76269934da4-hosts-file\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787357    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787370    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787382    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787399    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7f24485c-3402-4cef-84eb-6ea5dd63be64-cert\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787410    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a81cd320-aea5-4bbf-9a40-f83f814180b5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787437    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787463    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787465    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787487    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-kubelet-config\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787516    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787538    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787567    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-systemd-units\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787592    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-serviceca\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 13:00:21.787612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787614    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-run\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787626    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787647    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787675    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a81cd320-aea5-4bbf-9a40-f83f814180b5-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787676    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787677    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787611    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787772    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787824    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787862    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e3f637a5-1cd2-41a6-98d6-ad45904012ad-metrics-certs\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787886    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a81cd320-aea5-4bbf-9a40-f83f814180b5-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787908    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787917    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-tmp\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787932    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-kubelet-config\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787947    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-socket-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.787990    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788003    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-tmp\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788018    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-config\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788043    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztx68\" (UniqueName: \"kubernetes.io/projected/87328843-fa88-4d5d-9904-5a3036d5bb10-kube-api-access-ztx68\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788067    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788091    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-host\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788114    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-root\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788137    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788166    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788179    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57294d98-42b2-47f0-a79a-d9cf7053ff9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788235    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-host\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788280    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788261    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788319    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-serviceca\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788332    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788342    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788349    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-config\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788385    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-metrics-client-ca\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788483    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-client-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788489    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788496    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/57294d98-42b2-47f0-a79a-d9cf7053ff9c-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788507    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788596    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-system-cni-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788629    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788653    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788643    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-metrics-client-ca\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788682    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jgb\" (UniqueName: \"kubernetes.io/projected/448c900f-a9c5-4df1-b49e-e76269934da4-kube-api-access-55jgb\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788742    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-data-volume\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788769    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788794    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41aeee13-d6f6-4b22-b0d0-5c6a88038716-audit-log\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788819    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-systemd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788843    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788856    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-client-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788872    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c4a59dc-f603-4d34-a82a-23813a1b86b6-host-slash\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788879    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788898    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-tls\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788932    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788958    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-k8s-cni-cncf-io\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788966    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5c4a59dc-f603-4d34-a82a-23813a1b86b6-host-slash\") pod \"iptables-alerter-4p8sf\" (UID: \"5c4a59dc-f603-4d34-a82a-23813a1b86b6\") " pod="openshift-network-operator/iptables-alerter-4p8sf"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.788982    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-bin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789005    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-kubelet\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789030    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-kubelet\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789054    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-sys-fs\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789088    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789112    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789155    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/41aeee13-d6f6-4b22-b0d0-5c6a88038716-audit-log\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789231    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-data-volume\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789290    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-tls\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789343    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc748e5-08b1-4ffa-900c-09730777cc8d-config-volume\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789462    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-dbus\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789491    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3dc748e5-08b1-4ffa-900c-09730777cc8d-config-volume\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789495    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789522    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mc7\" (UniqueName: \"kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789539    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-wtmp\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789551    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/97d66a46-02bb-4220-a78c-850e615072ca-dbus\") pod \"global-pull-secret-syncer-c6hpz\" (UID: \"97d66a46-02bb-4220-a78c-850e615072ca\") " pod="kube-system/global-pull-secret-syncer-c6hpz"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789555    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-device-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789607    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789738    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-multus\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789775    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789798    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789813    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789832    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wj9\" (UniqueName: \"kubernetes.io/projected/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-kube-api-access-w4wj9\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789847    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-sys\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789867    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2frp7\" (UniqueName: \"kubernetes.io/projected/7f24485c-3402-4cef-84eb-6ea5dd63be64-kube-api-access-2frp7\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789873    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789891    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-tls\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789956    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789978    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-env-overrides\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.790031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.789997    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790069    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790118    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-tls\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790158    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790163    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-env-overrides\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790238    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-hostroot\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790314    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790350    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nwdg\" (UniqueName: \"kubernetes.io/projected/be0506ab-10a8-4158-b7e1-62ded5ce9f11-kube-api-access-9nwdg\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790377    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-bin\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790402    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-script-lib\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790415    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790429    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790459    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790584    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/686b38d0-13b9-4d2c-a1cf-5b8053804350-agent-certs\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790624    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc47f7a4-649c-4529-a608-4479486a59d8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790643    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790632    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjtw4\" (UniqueName: \"kubernetes.io/projected/3dc748e5-08b1-4ffa-900c-09730777cc8d-kube-api-access-xjtw4\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790681    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790719    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/87328843-fa88-4d5d-9904-5a3036d5bb10-ovnkube-script-lib\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790745    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-client-certs\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790776    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/686b38d0-13b9-4d2c-a1cf-5b8053804350-konnectivity-ca\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790804    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790833    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790860    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790863    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/686b38d0-13b9-4d2c-a1cf-5b8053804350-agent-certs\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790861    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790918    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790933    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/41aeee13-d6f6-4b22-b0d0-5c6a88038716-secret-metrics-server-client-certs\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790947    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4666\" (UniqueName: \"kubernetes.io/projected/bc47f7a4-649c-4529-a608-4479486a59d8-kube-api-access-q4666\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790959    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/686b38d0-13b9-4d2c-a1cf-5b8053804350-konnectivity-ca\") pod \"konnectivity-agent-phbg5\" (UID: \"686b38d0-13b9-4d2c-a1cf-5b8053804350\") " pod="kube-system/konnectivity-agent-phbg5"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.790976    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-registration-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791032    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8pj\" (UniqueName: \"kubernetes.io/projected/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kube-api-access-4h8pj\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791043    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791058    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791074    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791109    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-daemon-config\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791129    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791135    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791131    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-binary-copy\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791162    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-netd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791203    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-var-lib-kubelet\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791227    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-grpc-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791249    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791292    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-daemon-config\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791292    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-sys\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791318    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791328    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791331    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-thanos-querier-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791370    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48b5c\" (UniqueName: \"kubernetes.io/projected/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-kube-api-access-48b5c\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791451    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nhc\" (UniqueName: \"kubernetes.io/projected/41aeee13-d6f6-4b22-b0d0-5c6a88038716-kube-api-access-n2nhc\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791485    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-etc-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791529    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cnibin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791531    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-secret-grpc-tls\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791553    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-conf-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791600    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-textfile\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791665    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-accelerators-collector-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791697    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc748e5-08b1-4ffa-900c-09730777cc8d-tmp-dir\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791751    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-textfile\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791755    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-netns\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791830    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dc748e5-08b1-4ffa-900c-09730777cc8d-tmp-dir\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791836    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791878    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzj5\" (UniqueName: \"kubernetes.io/projected/5510e6da-dd62-4c2f-b73e-75ca74cad962-kube-api-access-lgzj5\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791940    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7k9\" (UniqueName: \"kubernetes.io/projected/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-api-access-zh7k9\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791983    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-accelerators-collector-config\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.791998    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792028    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792146    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cni-binary-copy\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792268    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6lbt\" (UniqueName: \"kubernetes.io/projected/e3f637a5-1cd2-41a6-98d6-ad45904012ad-kube-api-access-k6lbt\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 13:00:21.792251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792300    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792353    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dc748e5-08b1-4ffa-900c-09730777cc8d-metrics-tls\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792392    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc47f7a4-649c-4529-a608-4479486a59d8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792429    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cni-binary-copy\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792464    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792490    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprzm\" (UniqueName: \"kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792618    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3dc748e5-08b1-4ffa-900c-09730777cc8d-metrics-tls\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792620    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc47f7a4-649c-4529-a608-4479486a59d8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792635    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792652    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-metrics-server-audit-profiles\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792680    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792740    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87328843-fa88-4d5d-9904-5a3036d5bb10-ovn-node-metrics-cert\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792758    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792775    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792793    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-modprobe-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792816    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsq8j\" (UniqueName: \"kubernetes.io/projected/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-api-access-rsq8j\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792837    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792852    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-systemd\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792869    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792891    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-slash\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792915    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792931    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-multus-certs\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792946    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448c900f-a9c5-4df1-b49e-e76269934da4-tmp-dir\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792948    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/41aeee13-d6f6-4b22-b0d0-5c6a88038716-metrics-server-audit-profiles\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792964    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-metrics-client-ca\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.792984    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-socket-dir-parent\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793027    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-etc-kubernetes\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793045    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793185    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-node-log\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793229    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-system-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793245    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-cnibin\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793245    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793260    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-log-socket\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793298    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5510e6da-dd62-4c2f-b73e-75ca74cad962-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793311    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/448c900f-a9c5-4df1-b49e-e76269934da4-tmp-dir\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793309    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/87328843-fa88-4d5d-9904-5a3036d5bb10-ovn-node-metrics-cert\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793362    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793449    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793516    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.794841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.793520    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3814f51b-f30b-4f73-8061-ece6249a6edb-metrics-client-ca\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.797233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.797216    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.798562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.798547    5787 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:00:21.798605 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:21.798591    5787 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.798809 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.798796    5787 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 13:00:21.798843 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:21.798833    5787 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-83.us-east-2.compute.internal"
Oct 31 13:00:21.807344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.807319    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2frp7\" (UniqueName: \"kubernetes.io/projected/7f24485c-3402-4cef-84eb-6ea5dd63be64-kube-api-access-2frp7\") pod \"ingress-canary-2798k\" (UID: \"7f24485c-3402-4cef-84eb-6ea5dd63be64\") " pod="openshift-ingress-canary/ingress-canary-2798k"
Oct 31 13:00:21.809672 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.809650    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jgb\" (UniqueName: \"kubernetes.io/projected/448c900f-a9c5-4df1-b49e-e76269934da4-kube-api-access-55jgb\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 13:00:21.810061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.810043    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gztt\" (UniqueName: \"kubernetes.io/projected/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-kube-api-access-4gztt\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.810218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.810200    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nhc\" (UniqueName: \"kubernetes.io/projected/41aeee13-d6f6-4b22-b0d0-5c6a88038716-kube-api-access-n2nhc\") pod \"metrics-server-b9f69b5fb-mnd5t\" (UID: \"41aeee13-d6f6-4b22-b0d0-5c6a88038716\") " pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:21.810264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.810244    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48b5c\" (UniqueName: \"kubernetes.io/projected/2bcc9522-04bc-46ec-873c-ac4d00b96ffb-kube-api-access-48b5c\") pod \"thanos-querier-7d6b4cbd64-96gzv\" (UID: \"2bcc9522-04bc-46ec-873c-ac4d00b96ffb\") " pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:21.810929 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.810913    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4666\" (UniqueName: \"kubernetes.io/projected/bc47f7a4-649c-4529-a608-4479486a59d8-kube-api-access-q4666\") pod \"openshift-state-metrics-7bd7489957-t8wbk\" (UID: \"bc47f7a4-649c-4529-a608-4479486a59d8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-t8wbk"
Oct 31 13:00:21.812464 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.812450    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6lbt\" (UniqueName: \"kubernetes.io/projected/e3f637a5-1cd2-41a6-98d6-ad45904012ad-kube-api-access-k6lbt\") pod \"network-metrics-daemon-jbb9m\" (UID: \"e3f637a5-1cd2-41a6-98d6-ad45904012ad\") " pod="openshift-multus/network-metrics-daemon-jbb9m"
Oct 31 13:00:21.813877 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.813861    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8pj\" (UniqueName: \"kubernetes.io/projected/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kube-api-access-4h8pj\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.814059 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.814044    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2n6k\" (UniqueName: \"kubernetes.io/projected/3814f51b-f30b-4f73-8061-ece6249a6edb-kube-api-access-s2n6k\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.814624 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.814606    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mc7\" (UniqueName: \"kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7\") pod \"console-64cd84c7c9-bhh7z\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") " pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:21.814722 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.814693    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx59n\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n\") pod \"alertmanager-main-0\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:21.815567 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.815551    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztx68\" (UniqueName: \"kubernetes.io/projected/87328843-fa88-4d5d-9904-5a3036d5bb10-kube-api-access-ztx68\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.815671 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.815652    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rxv\" (UniqueName: \"kubernetes.io/projected/a81cd320-aea5-4bbf-9a40-f83f814180b5-kube-api-access-w8rxv\") pod \"prometheus-operator-9cd89b74c-r8ln9\" (UID: \"a81cd320-aea5-4bbf-9a40-f83f814180b5\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-r8ln9"
Oct 31 13:00:21.815717 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.815657    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6db7\" (UniqueName: \"kubernetes.io/projected/3c98a803-2cc2-4806-8a14-b05c31ee7f03-kube-api-access-t6db7\") pod \"network-check-target-r4bj6\" (UID: \"3c98a803-2cc2-4806-8a14-b05c31ee7f03\") " pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 13:00:21.824693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.824663    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zkqv\" (UniqueName: \"kubernetes.io/projected/1e7e4656-607a-4995-a250-a3bda30be0c1-kube-api-access-9zkqv\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.824693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.824723    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wj9\" (UniqueName: \"kubernetes.io/projected/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-kube-api-access-w4wj9\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 13:00:21.824916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.824663    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsq8j\" (UniqueName: \"kubernetes.io/projected/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-kube-api-access-rsq8j\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.824916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.824780    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjtw4\" (UniqueName: \"kubernetes.io/projected/3dc748e5-08b1-4ffa-900c-09730777cc8d-kube-api-access-xjtw4\") pod \"dns-default-wwg8z\" (UID: \"3dc748e5-08b1-4ffa-900c-09730777cc8d\") " pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:21.824916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.824840    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzj5\" (UniqueName: \"kubernetes.io/projected/5510e6da-dd62-4c2f-b73e-75ca74cad962-kube-api-access-lgzj5\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.824916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.824841    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7k9\" (UniqueName: \"kubernetes.io/projected/57294d98-42b2-47f0-a79a-d9cf7053ff9c-kube-api-access-zh7k9\") pod \"kube-state-metrics-5dc8d7d89b-2xmtm\" (UID: \"57294d98-42b2-47f0-a79a-d9cf7053ff9c\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2xmtm"
Oct 31 13:00:21.827274 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.827249    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nwdg\" (UniqueName: \"kubernetes.io/projected/be0506ab-10a8-4158-b7e1-62ded5ce9f11-kube-api-access-9nwdg\") pod \"monitoring-plugin-77f4f76594-sqfbr\" (UID: \"be0506ab-10a8-4158-b7e1-62ded5ce9f11\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:21.837657 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.837633    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprzm\" (UniqueName: \"kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm\") pod \"console-f4dfdf7b8-7hlkd\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") " pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:21.893869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893840    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.893869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893871    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-k8s-cni-cncf-io\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893892    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-bin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893916    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-kubelet\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893937    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-kubelet\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893956    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-sys-fs\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893980    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-wtmp\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893988    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-kubelet\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894001    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-device-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893988    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-bin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894028    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-multus\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894043    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-sys-fs\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894056    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-sys\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894047    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894068    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-device-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894069    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-node-exporter-wtmp\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.893976    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-k8s-cni-cncf-io\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894082    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-var-lib-cni-multus\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894105    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-hostroot\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894116    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-sys\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894128    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-bin\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894149    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894155    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-hostroot\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894162    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-bin\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894168    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-kubelet\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.894172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894166    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-registration-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894212    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-registration-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894228    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-netd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894258    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-cni-netd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894230    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894295    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-var-lib-kubelet\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894258    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-var-lib-kubelet\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894326    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-sys\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894355    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-etc-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894371    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cnibin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894392    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-conf-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894416    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-etc-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894430    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-cnibin\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894420    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-netns\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894448    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-netns\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894396    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-sys\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894473    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894476    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-conf-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894499    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894504    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894517    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-modprobe-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894534    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894539    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894551    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-systemd\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894566    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-slash\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894582    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-multus-certs\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894597    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-socket-dir-parent\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894608    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-slash\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894616    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894614    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-etc-kubernetes\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894646    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-node-log\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894611    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-systemd\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894647    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-host-run-multus-certs\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894645    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-etc-kubernetes\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894669    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-multus-socket-dir-parent\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894676    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-node-log\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894683    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-modprobe-d\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894673    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-system-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894735    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5337ff71-42fc-4f7d-a92f-67e13f9c0a00-system-cni-dir\") pod \"multus-7pzvw\" (UID: \"5337ff71-42fc-4f7d-a92f-67e13f9c0a00\") " pod="openshift-multus/multus-7pzvw"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894740    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-cnibin\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894756    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-log-socket\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894771    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894773    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-cnibin\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894791    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-conf\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894798    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-log-socket\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894811    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-ovn\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894827    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-ovn-kubernetes\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894830    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysconfig\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894849    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-lib-modules\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894855    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysconfig\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894867    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-crio-socket\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894885    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-netns\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894892    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-sysctl-conf\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894889    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-ovn\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894909    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-var-lib-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894914    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/8b3fbffa-ba33-46db-9bdf-fcdf578955f6-crio-socket\") pod \"insights-runtime-extractor-nkxs6\" (UID: \"8b3fbffa-ba33-46db-9bdf-fcdf578955f6\") " pod="openshift-insights/insights-runtime-extractor-nkxs6"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894931    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-host-run-netns\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894943    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-kubernetes\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894954    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-var-lib-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894965    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894979    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-lib-modules\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.894992    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/448c900f-a9c5-4df1-b49e-e76269934da4-hosts-file\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895008    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-etc-kubernetes\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895023    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-systemd-units\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895028    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-etc-selinux\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895048    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-run\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895054    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/448c900f-a9c5-4df1-b49e-e76269934da4-hosts-file\") pod \"node-resolver-ld8vz\" (UID: \"448c900f-a9c5-4df1-b49e-e76269934da4\") " pod="openshift-dns/node-resolver-ld8vz"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895067    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-systemd-units\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895075    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-socket-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895099    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-host\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 13:00:21.895076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895113    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-root\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895128    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-host\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895147    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e22c3df7-5789-4439-ba22-3a7d2b30c3fa-host\") pod \"node-ca-lvtzl\" (UID: \"e22c3df7-5789-4439-ba22-3a7d2b30c3fa\") " pod="openshift-image-registry/node-ca-lvtzl"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895151    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b16b0cb-c872-4fee-8a2a-83adb2777cc6-socket-dir\") pod \"aws-ebs-csi-driver-node-g2bsb\" (UID: \"5b16b0cb-c872-4fee-8a2a-83adb2777cc6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-g2bsb"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895151    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-system-cni-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895172    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3814f51b-f30b-4f73-8061-ece6249a6edb-root\") pod \"node-exporter-srtvx\" (UID: \"3814f51b-f30b-4f73-8061-ece6249a6edb\") " pod="openshift-monitoring/node-exporter-srtvx"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895185    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-systemd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895188    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5510e6da-dd62-4c2f-b73e-75ca74cad962-system-cni-dir\") pod \"multus-additional-cni-plugins-6rkkf\" (UID: \"5510e6da-dd62-4c2f-b73e-75ca74cad962\") " pod="openshift-multus/multus-additional-cni-plugins-6rkkf"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895188    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-host\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895200    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e7e4656-607a-4995-a250-a3bda30be0c1-run\") pod \"tuned-5pwr7\" (UID: \"1e7e4656-607a-4995-a250-a3bda30be0c1\") " pod="openshift-cluster-node-tuning-operator/tuned-5pwr7"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895204    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895230    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-systemd\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.897262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.895230    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/87328843-fa88-4d5d-9904-5a3036d5bb10-run-openvswitch\") pod \"ovnkube-node-cbknb\" (UID: \"87328843-fa88-4d5d-9904-5a3036d5bb10\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:21.924448 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:21.924361    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f4dfdf7b8-7hlkd"]
Oct 31 13:00:22.634665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:22.634634    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:23.172818 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:23.172790    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wwg8z"
Oct 31 13:00:23.323880 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:23.322564    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:24.403363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:24.403330    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:24.406900 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:24.406870    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-sqfbr"
Oct 31 13:00:24.629944 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:24.629909    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:25.545832 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:25.545789    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:25.551398 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:25.551372    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d6b4cbd64-96gzv"
Oct 31 13:00:27.977110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:27.977073    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:29.087359 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:29.087323    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 13:00:30.050774 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:30.050746    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:30.051042 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:30.050959    5787 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 13:00:34.634682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:34.634650    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:34.638476 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:34.638453    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:00:37.266402 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:37.266373    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-phbg5"
Oct 31 13:00:38.998937 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:38.998909    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbknb"
Oct 31 13:00:40.717323 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:40.717298    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:40.721367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:40.721350    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:41.665471 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:41.665437    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:00:47.335859 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.335824    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 13:00:47.336290 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.336139    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="alertmanager" containerID="cri-o://88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f" gracePeriod=120
Oct 31 13:00:47.336290 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.336191    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy-metric" containerID="cri-o://470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40" gracePeriod=120
Oct 31 13:00:47.336290 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.336248    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="config-reloader" containerID="cri-o://862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20" gracePeriod=120
Oct 31 13:00:47.336483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.336209    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy-web" containerID="cri-o://8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034" gracePeriod=120
Oct 31 13:00:47.336483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.336228    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy" containerID="cri-o://7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13" gracePeriod=120
Oct 31 13:00:47.336483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.336280    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="prom-label-proxy" containerID="cri-o://18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92" gracePeriod=120
Oct 31 13:00:47.800893 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.800856    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f4dfdf7b8-7hlkd" podUID="e505bf1f-b560-4226-9a6e-fad97121cbfe" containerName="console" containerID="cri-o://795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c" gracePeriod=15
Oct 31 13:00:47.875347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875318    5787 generic.go:356] "Generic (PLEG): container finished" podID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerID="18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92" exitCode=0
Oct 31 13:00:47.875347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875341    5787 generic.go:356] "Generic (PLEG): container finished" podID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerID="470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40" exitCode=0
Oct 31 13:00:47.875347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875348    5787 generic.go:356] "Generic (PLEG): container finished" podID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerID="7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13" exitCode=0
Oct 31 13:00:47.875347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875353    5787 generic.go:356] "Generic (PLEG): container finished" podID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerID="862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20" exitCode=0
Oct 31 13:00:47.875668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875361    5787 generic.go:356] "Generic (PLEG): container finished" podID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerID="88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f" exitCode=0
Oct 31 13:00:47.875668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875394    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerDied","Data":"18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92"}
Oct 31 13:00:47.875668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875471    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerDied","Data":"470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40"}
Oct 31 13:00:47.875668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875508    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerDied","Data":"7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13"}
Oct 31 13:00:47.875668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875547    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerDied","Data":"862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20"}
Oct 31 13:00:47.875668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:47.875587    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerDied","Data":"88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f"}
Oct 31 13:00:48.044820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.044795    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:48.232036 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232007    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm\" (UniqueName: \"kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm\") pod \"e505bf1f-b560-4226-9a6e-fad97121cbfe\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") "
Oct 31 13:00:48.232233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232059    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert\") pod \"e505bf1f-b560-4226-9a6e-fad97121cbfe\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") "
Oct 31 13:00:48.232233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232085    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert\") pod \"e505bf1f-b560-4226-9a6e-fad97121cbfe\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") "
Oct 31 13:00:48.232233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232114    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config\") pod \"e505bf1f-b560-4226-9a6e-fad97121cbfe\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") "
Oct 31 13:00:48.232233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232147    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config\") pod \"e505bf1f-b560-4226-9a6e-fad97121cbfe\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") "
Oct 31 13:00:48.232233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232179    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp\") pod \"e505bf1f-b560-4226-9a6e-fad97121cbfe\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") "
Oct 31 13:00:48.232233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232214    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca\") pod \"e505bf1f-b560-4226-9a6e-fad97121cbfe\" (UID: \"e505bf1f-b560-4226-9a6e-fad97121cbfe\") "
Oct 31 13:00:48.232546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232517    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert" pod "e505bf1f-b560-4226-9a6e-fad97121cbfe" (UID: "e505bf1f-b560-4226-9a6e-fad97121cbfe"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:48.232689 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232660    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca" pod "e505bf1f-b560-4226-9a6e-fad97121cbfe" (UID: "e505bf1f-b560-4226-9a6e-fad97121cbfe"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:48.232923 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232900    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp" pod "e505bf1f-b560-4226-9a6e-fad97121cbfe" (UID: "e505bf1f-b560-4226-9a6e-fad97121cbfe"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:48.233004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.232982    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config" pod "e505bf1f-b560-4226-9a6e-fad97121cbfe" (UID: "e505bf1f-b560-4226-9a6e-fad97121cbfe"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:48.234828 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.234799    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config" pod "e505bf1f-b560-4226-9a6e-fad97121cbfe" (UID: "e505bf1f-b560-4226-9a6e-fad97121cbfe"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:48.234957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.234893    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert" pod "e505bf1f-b560-4226-9a6e-fad97121cbfe" (UID: "e505bf1f-b560-4226-9a6e-fad97121cbfe"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:48.235013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.234996    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm" pod "e505bf1f-b560-4226-9a6e-fad97121cbfe" (UID: "e505bf1f-b560-4226-9a6e-fad97121cbfe"). InnerVolumeSpecName "kube-api-access-qprzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:48.247206 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:48.247170    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:00:48.333411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.333376    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qprzm\" (UniqueName: \"kubernetes.io/projected/e505bf1f-b560-4226-9a6e-fad97121cbfe-kube-api-access-qprzm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.333411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.333405    5787 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-oauth-serving-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.333411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.333418    5787 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-serving-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.333643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.333428    5787 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-config\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.333643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.333439    5787 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e505bf1f-b560-4226-9a6e-fad97121cbfe-console-oauth-config\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.333643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.333448    5787 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e505bf1f-b560-4226-9a6e-fad97121cbfe-tmp\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.333643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.333457    5787 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e505bf1f-b560-4226-9a6e-fad97121cbfe-service-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.576586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.576562    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:48.737265 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737225    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737274    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737306    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737336    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737384    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737411    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737462    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737531    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737579    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737607    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737635    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737666    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.737740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737692    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config\") pod \"2ab4d241-bf18-44f2-a803-1de48ef9086e\" (UID: \"2ab4d241-bf18-44f2-a803-1de48ef9086e\") "
Oct 31 13:00:48.738093 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737685    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:48.738093 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737920    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:00:48.738093 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.737976    5787 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.738970 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.738933    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:48.741301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.741276    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:48.741456 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.741299    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:48.741551 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.741474    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:48.741722 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.741642    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:48.741794 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.741737    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:48.741866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.741847    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:48.742363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.742322    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:00:48.742452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.742374    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "kube-api-access-fx59n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:00:48.745944 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.745922    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:48.752266 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.752242    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config" pod "2ab4d241-bf18-44f2-a803-1de48ef9086e" (UID: "2ab4d241-bf18-44f2-a803-1de48ef9086e"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:00:48.838685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838601    5787 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.838685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838633    5787 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-out\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.838685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838644    5787 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-web-config\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.838685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838656    5787 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-main-tls\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.838685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838666    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx59n\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-kube-api-access-fx59n\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.838685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838675    5787 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.838685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838685    5787 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2ab4d241-bf18-44f2-a803-1de48ef9086e-alertmanager-main-db\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.839061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838695    5787 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-config-volume\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.839061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838724    5787 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ab4d241-bf18-44f2-a803-1de48ef9086e-metrics-client-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.839061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838732    5787 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-cluster-tls-config\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.839061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838742    5787 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2ab4d241-bf18-44f2-a803-1de48ef9086e-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.839061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.838751    5787 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2ab4d241-bf18-44f2-a803-1de48ef9086e-tls-assets\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:00:48.884103 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.884064    5787 generic.go:356] "Generic (PLEG): container finished" podID="e505bf1f-b560-4226-9a6e-fad97121cbfe" containerID="795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c" exitCode=0
Oct 31 13:00:48.884249 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.884134    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4dfdf7b8-7hlkd"
Oct 31 13:00:48.884249 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.884148    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4dfdf7b8-7hlkd" event={"ID":"e505bf1f-b560-4226-9a6e-fad97121cbfe","Type":"ContainerDied","Data":"795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c"}
Oct 31 13:00:48.884249 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.884240    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4dfdf7b8-7hlkd" event={"ID":"e505bf1f-b560-4226-9a6e-fad97121cbfe","Type":"ContainerDied","Data":"1b45337793c79920fff6e52e54fb47fb4d48eb004fa6b5c94a75d0410e99a507"}
Oct 31 13:00:48.884403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.884266    5787 scope.go:122] "RemoveContainer" containerID="795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c"
Oct 31 13:00:48.889383 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.889351    5787 generic.go:356] "Generic (PLEG): container finished" podID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerID="8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034" exitCode=0
Oct 31 13:00:48.889548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.889460    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerDied","Data":"8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034"}
Oct 31 13:00:48.889548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.889511    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:48.889548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.889533    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2ab4d241-bf18-44f2-a803-1de48ef9086e","Type":"ContainerDied","Data":"0736cae6a049b56e1016fcb9045064f8a3cb4b2cfd3ec2fb4c7973a886d7627a"}
Oct 31 13:00:48.895863 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.895837    5787 scope.go:122] "RemoveContainer" containerID="795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c"
Oct 31 13:00:48.896224 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:48.896196    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c\": container with ID starting with 795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c not found: ID does not exist" containerID="795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c"
Oct 31 13:00:48.896338 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.896229    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c"} err="failed to get container status \"795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c\": rpc error: code = NotFound desc = could not find container \"795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c\": container with ID starting with 795905901d8d17ead51a30be7c6678eb04502e92eda524e6e9845f23d221ef9c not found: ID does not exist"
Oct 31 13:00:48.896338 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.896264    5787 scope.go:122] "RemoveContainer" containerID="18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92"
Oct 31 13:00:48.905475 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.905450    5787 scope.go:122] "RemoveContainer" containerID="470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40"
Oct 31 13:00:48.905588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.905451    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f4dfdf7b8-7hlkd"]
Oct 31 13:00:48.910072 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.910044    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f4dfdf7b8-7hlkd"]
Oct 31 13:00:48.913882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.913861    5787 scope.go:122] "RemoveContainer" containerID="7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13"
Oct 31 13:00:48.922367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.922343    5787 scope.go:122] "RemoveContainer" containerID="8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034"
Oct 31 13:00:48.929923 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.929901    5787 scope.go:122] "RemoveContainer" containerID="862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20"
Oct 31 13:00:48.932402 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.932377    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 13:00:48.937284 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.937247    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 13:00:48.939427 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.939410    5787 scope.go:122] "RemoveContainer" containerID="88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f"
Oct 31 13:00:48.947728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.947689    5787 scope.go:122] "RemoveContainer" containerID="b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055"
Oct 31 13:00:48.955861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.955831    5787 scope.go:122] "RemoveContainer" containerID="18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92"
Oct 31 13:00:48.956183 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:48.956163    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92\": container with ID starting with 18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92 not found: ID does not exist" containerID="18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92"
Oct 31 13:00:48.956264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.956195    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92"} err="failed to get container status \"18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92\": rpc error: code = NotFound desc = could not find container \"18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92\": container with ID starting with 18ddfe3e25e9200c3db602bace8359177dc6676e4dc8cd2f6835853f69a81d92 not found: ID does not exist"
Oct 31 13:00:48.956264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.956222    5787 scope.go:122] "RemoveContainer" containerID="470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40"
Oct 31 13:00:48.956480 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:48.956461    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40\": container with ID starting with 470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40 not found: ID does not exist" containerID="470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40"
Oct 31 13:00:48.956523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.956488    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40"} err="failed to get container status \"470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40\": rpc error: code = NotFound desc = could not find container \"470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40\": container with ID starting with 470a926e13bfae31ef0d4b6c7277315ca4db49b5d836e2cb533aedc838e5cb40 not found: ID does not exist"
Oct 31 13:00:48.956523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.956506    5787 scope.go:122] "RemoveContainer" containerID="7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13"
Oct 31 13:00:48.956810 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:48.956793    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13\": container with ID starting with 7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13 not found: ID does not exist" containerID="7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13"
Oct 31 13:00:48.956881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.956817    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13"} err="failed to get container status \"7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13\": rpc error: code = NotFound desc = could not find container \"7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13\": container with ID starting with 7c2b0d6f830b96073351b396fb3ba83677334df318ba82f462e8e627afddab13 not found: ID does not exist"
Oct 31 13:00:48.956881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.956836    5787 scope.go:122] "RemoveContainer" containerID="8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034"
Oct 31 13:00:48.957094 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:48.957076    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034\": container with ID starting with 8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034 not found: ID does not exist" containerID="8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034"
Oct 31 13:00:48.957146 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.957100    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034"} err="failed to get container status \"8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034\": rpc error: code = NotFound desc = could not find container \"8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034\": container with ID starting with 8bceb8adb99e6fdfa4c9a94a918d65c61deda441bf5b2f54139b60cf10783034 not found: ID does not exist"
Oct 31 13:00:48.957146 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.957114    5787 scope.go:122] "RemoveContainer" containerID="862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20"
Oct 31 13:00:48.957353 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:48.957334    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20\": container with ID starting with 862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20 not found: ID does not exist" containerID="862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20"
Oct 31 13:00:48.957412 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.957360    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20"} err="failed to get container status \"862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20\": rpc error: code = NotFound desc = could not find container \"862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20\": container with ID starting with 862d32bfc8c058c61d318a6d73034021ca837b70bd863235d4c1e37679c29b20 not found: ID does not exist"
Oct 31 13:00:48.957412 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.957378    5787 scope.go:122] "RemoveContainer" containerID="88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f"
Oct 31 13:00:48.957604 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:48.957586    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f\": container with ID starting with 88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f not found: ID does not exist" containerID="88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f"
Oct 31 13:00:48.957643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.957607    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f"} err="failed to get container status \"88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f\": rpc error: code = NotFound desc = could not find container \"88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f\": container with ID starting with 88492cf6f1cabcbe2cb5a4a9238490fcb40c6ae5a3bbf0783c398a813b7a188f not found: ID does not exist"
Oct 31 13:00:48.957643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.957619    5787 scope.go:122] "RemoveContainer" containerID="b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055"
Oct 31 13:00:48.957841 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:00:48.957823    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055\": container with ID starting with b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055 not found: ID does not exist" containerID="b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055"
Oct 31 13:00:48.957899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.957851    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055"} err="failed to get container status \"b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055\": rpc error: code = NotFound desc = could not find container \"b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055\": container with ID starting with b4205dc501894a5fdf74fa2ca66b1ec4f8f6c9a10628210c68f07f0f625e4055 not found: ID does not exist"
Oct 31 13:00:48.975012 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.974980    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 13:00:48.975262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975251    5787 cpu_manager.go:414] "containerMap: removing container" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy-web"
Oct 31 13:00:48.975304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975263    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy-web"
Oct 31 13:00:48.975304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975273    5787 cpu_manager.go:414] "containerMap: removing container" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="config-reloader"
Oct 31 13:00:48.975304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975278    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="config-reloader"
Oct 31 13:00:48.975304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975291    5787 cpu_manager.go:414] "containerMap: removing container" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy-metric"
Oct 31 13:00:48.975304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975296    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy-metric"
Oct 31 13:00:48.975304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975303    5787 cpu_manager.go:414] "containerMap: removing container" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="alertmanager"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975308    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="alertmanager"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975320    5787 cpu_manager.go:414] "containerMap: removing container" podUID="e505bf1f-b560-4226-9a6e-fad97121cbfe" containerName="console"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975325    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e505bf1f-b560-4226-9a6e-fad97121cbfe" containerName="console"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975335    5787 cpu_manager.go:414] "containerMap: removing container" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975340    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975350    5787 cpu_manager.go:414] "containerMap: removing container" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="init-config-reloader"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975354    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="init-config-reloader"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975360    5787 cpu_manager.go:414] "containerMap: removing container" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="prom-label-proxy"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975364    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="prom-label-proxy"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975402    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="config-reloader"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975409    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy-metric"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975416    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="alertmanager"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975422    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="e505bf1f-b560-4226-9a6e-fad97121cbfe" containerName="console"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975428    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975433    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="init-config-reloader"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975438    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="prom-label-proxy"
Oct 31 13:00:48.975480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.975444    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" containerName="kube-rbac-proxy-web"
Oct 31 13:00:48.979008 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.978988    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:48.984004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.983961    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 31 13:00:48.984004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.983968    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 31 13:00:48.984004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.983968    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 31 13:00:48.984004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.983969    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 31 13:00:48.984004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.983968    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 31 13:00:48.984362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.984102    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 31 13:00:48.984362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.984148    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 31 13:00:48.984362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.984158    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jfvtg\""
Oct 31 13:00:48.984362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.984200    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 31 13:00:48.988218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.988196    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 31 13:00:48.997852 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:48.997820    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 13:00:49.040557 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040521    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.040557 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040563    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.040834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040584    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-config-out\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.040834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040652    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-web-config\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.040834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040717    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.040834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040784    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.040834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040831    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.041028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040854    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmz8z\" (UniqueName: \"kubernetes.io/projected/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-kube-api-access-zmz8z\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.041028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040874    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.041028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.040890    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.041028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.041022    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.041165 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.041060    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.041165 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.041085    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.090068 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.089993    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-r4bj6"
Oct 31 13:00:49.141585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141550    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.141778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141605    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.141778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141641    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.141778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141693    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.141778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141746    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.141778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141772    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-config-out\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.142065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141796    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-web-config\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.142065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141833    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.142065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141859    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.142065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141886    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.142065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141917    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmz8z\" (UniqueName: \"kubernetes.io/projected/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-kube-api-access-zmz8z\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.142065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141947    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.142065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.141970    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.142065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.142004    5787 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 13:00:49.142799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.142474    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.142799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.142670    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.143782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.143728    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.145528 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.145504    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.146409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.145993    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.146409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.146021    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-web-config\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.146409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.146181    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.146409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.146375    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-config-out\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.146661 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.146639    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.146772 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.146752    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.146846 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.146831    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.147088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.147073    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.156463 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.156437    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmz8z\" (UniqueName: \"kubernetes.io/projected/0d86fc01-6e17-4d9d-9e6d-eac5fe617705-kube-api-access-zmz8z\") pod \"alertmanager-main-0\" (UID: \"0d86fc01-6e17-4d9d-9e6d-eac5fe617705\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.300627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.300589    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 31 13:00:49.589107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.589082    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 31 13:00:49.590965 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:49.590929    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d86fc01_6e17_4d9d_9e6d_eac5fe617705.slice/crio-9aac280150e93ab1d8d480eb4b95abe2fa89367c333aafc78625e0790fd18066 WatchSource:0}: Error finding container 9aac280150e93ab1d8d480eb4b95abe2fa89367c333aafc78625e0790fd18066: Status 404 returned error can't find the container with id 9aac280150e93ab1d8d480eb4b95abe2fa89367c333aafc78625e0790fd18066
Oct 31 13:00:49.895339 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.895267    5787 generic.go:356] "Generic (PLEG): container finished" podID="0d86fc01-6e17-4d9d-9e6d-eac5fe617705" containerID="e69a0cfa620d2591b21e3d4812a39c0ad6c16911931cafb8f7b7f38a398a2634" exitCode=0
Oct 31 13:00:49.895498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.895354    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d86fc01-6e17-4d9d-9e6d-eac5fe617705","Type":"ContainerDied","Data":"e69a0cfa620d2591b21e3d4812a39c0ad6c16911931cafb8f7b7f38a398a2634"}
Oct 31 13:00:49.895498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:49.895441    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d86fc01-6e17-4d9d-9e6d-eac5fe617705","Type":"ContainerStarted","Data":"9aac280150e93ab1d8d480eb4b95abe2fa89367c333aafc78625e0790fd18066"}
Oct 31 13:00:50.703779 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:50.703739    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ab4d241-bf18-44f2-a803-1de48ef9086e" path="/var/lib/kubelet/pods/2ab4d241-bf18-44f2-a803-1de48ef9086e/volumes"
Oct 31 13:00:50.704468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:50.704454    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e505bf1f-b560-4226-9a6e-fad97121cbfe" path="/var/lib/kubelet/pods/e505bf1f-b560-4226-9a6e-fad97121cbfe/volumes"
Oct 31 13:00:50.904913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:50.904881    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d86fc01-6e17-4d9d-9e6d-eac5fe617705","Type":"ContainerStarted","Data":"1f67cc5daf34f53dc6d4950b9890d75d915157c5cd7ca25c934887d8db5528fc"}
Oct 31 13:00:50.905097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:50.904945    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d86fc01-6e17-4d9d-9e6d-eac5fe617705","Type":"ContainerStarted","Data":"82b25c39f772709e381e337b291c9d0ddad551482308ca825916c80d065abcd3"}
Oct 31 13:00:50.905097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:50.904972    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d86fc01-6e17-4d9d-9e6d-eac5fe617705","Type":"ContainerStarted","Data":"c9036220a3f21d24905cb7a022b6f5306beb7690bae8cd4e3ae71d3387884e81"}
Oct 31 13:00:50.905097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:50.904997    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d86fc01-6e17-4d9d-9e6d-eac5fe617705","Type":"ContainerStarted","Data":"bee96c050279774e6818dd6d6f0194ae04f6f0dfa7e0cafeaa7529bf6bb4161f"}
Oct 31 13:00:50.905097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:50.905018    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d86fc01-6e17-4d9d-9e6d-eac5fe617705","Type":"ContainerStarted","Data":"9455652fcafd4f89c040ae48242f5812e769fa866472c925a4815dc1734b563f"}
Oct 31 13:00:50.905097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:50.905074    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d86fc01-6e17-4d9d-9e6d-eac5fe617705","Type":"ContainerStarted","Data":"1d3d5af1fd82657aa0157624e61cb8cb8a67ad6f9ef2759a09fd93865d8c2248"}
Oct 31 13:00:50.941728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:50.940417    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.927939834 podStartE2EDuration="2.940397695s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.457861ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:50.938735762 +0000 UTC m=+30.354677604" watchObservedRunningTime="2026-10-31 13:00:50.940397695 +0000 UTC m=+30.356339535"
Oct 31 13:00:51.350271 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.350235    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"]
Oct 31 13:00:51.354521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.354495    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.360376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.359551    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 31 13:00:51.360376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.359568    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 31 13:00:51.360376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.359623    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 31 13:00:51.360376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.359735    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-85s4l\""
Oct 31 13:00:51.360376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.359751    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 31 13:00:51.360376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.359799    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-secret-telemeter-client\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.360376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.359880    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 31 13:00:51.360857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.360621    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b46d9c-9428-4785-b432-0765ce8ea880-metrics-client-ca\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.360910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.360850    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.361080 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.360918    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsvmd\" (UniqueName: \"kubernetes.io/projected/85b46d9c-9428-4785-b432-0765ce8ea880-kube-api-access-bsvmd\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.361210 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.361112    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b46d9c-9428-4785-b432-0765ce8ea880-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.361372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.361235    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-telemeter-client-tls\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.361438 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.361393    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-federate-client-tls\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.361533 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.361511    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b46d9c-9428-4785-b432-0765ce8ea880-serving-certs-ca-bundle\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.366676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.366628    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 31 13:00:51.373305 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.373214    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"]
Oct 31 13:00:51.464881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.464836    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b46d9c-9428-4785-b432-0765ce8ea880-serving-certs-ca-bundle\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.465075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.464901    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-secret-telemeter-client\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.465075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.464926    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b46d9c-9428-4785-b432-0765ce8ea880-metrics-client-ca\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.465075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.464960    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.465075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.465004    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsvmd\" (UniqueName: \"kubernetes.io/projected/85b46d9c-9428-4785-b432-0765ce8ea880-kube-api-access-bsvmd\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.465075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.465030    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b46d9c-9428-4785-b432-0765ce8ea880-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.465075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.465050    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-telemeter-client-tls\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.465075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.465065    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-federate-client-tls\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.465738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.465645    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b46d9c-9428-4785-b432-0765ce8ea880-serving-certs-ca-bundle\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.465738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.465665    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85b46d9c-9428-4785-b432-0765ce8ea880-metrics-client-ca\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.466166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.466140    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85b46d9c-9428-4785-b432-0765ce8ea880-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.467691 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.467665    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-federate-client-tls\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.467934 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.467910    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-telemeter-client-tls\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.468118 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.468102    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.468176 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.468158    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/85b46d9c-9428-4785-b432-0765ce8ea880-secret-telemeter-client\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.479626 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.479596    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsvmd\" (UniqueName: \"kubernetes.io/projected/85b46d9c-9428-4785-b432-0765ce8ea880-kube-api-access-bsvmd\") pod \"telemeter-client-7dd9dd9b4d-zj579\" (UID: \"85b46d9c-9428-4785-b432-0765ce8ea880\") " pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.676315 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.676158    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"
Oct 31 13:00:51.957026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.956983    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579"]
Oct 31 13:00:51.964388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:51.964364    5787 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:00:52.914099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:52.914066    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579" event={"ID":"85b46d9c-9428-4785-b432-0765ce8ea880","Type":"ContainerStarted","Data":"c4edd10c1d25210fdb07b762350d3df7e360bdd8959de6e7c35fbc69709156bb"}
Oct 31 13:00:54.924721 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:54.924660    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579" event={"ID":"85b46d9c-9428-4785-b432-0765ce8ea880","Type":"ContainerStarted","Data":"28e4875dae530f5e6dfab7ad0e447111379588782b8d02f02324413437edc145"}
Oct 31 13:00:54.925146 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:54.924744    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579" event={"ID":"85b46d9c-9428-4785-b432-0765ce8ea880","Type":"ContainerStarted","Data":"7770a7185747e055179fd055af1d2b3e6aa319f2e272468a9f0031d7f5464cbb"}
Oct 31 13:00:54.925146 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:54.924766    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579" event={"ID":"85b46d9c-9428-4785-b432-0765ce8ea880","Type":"ContainerStarted","Data":"681417b2cd7463d7ce933929c5686fb86df27cce549f5414bd1b6ca335b4fcb2"}
Oct 31 13:00:54.959892 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:54.959837    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7dd9dd9b4d-zj579" podStartSLOduration=1.663901633 podStartE2EDuration="3.959820818s" totalImagesPullingTime="2.295919185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:54.95792007 +0000 UTC m=+34.373861910" watchObservedRunningTime="2026-10-31 13:00:54.959820818 +0000 UTC m=+34.375762649"
Oct 31 13:00:55.064621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.064577    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-dfd6fc47c-vkqnx"]
Oct 31 13:00:55.067120 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.067095    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.082472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.082433    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dfd6fc47c-vkqnx"]
Oct 31 13:00:55.099388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.099354    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-trusted-ca-bundle\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.099388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.099396    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97ngj\" (UniqueName: \"kubernetes.io/projected/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-kube-api-access-97ngj\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.099605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.099417    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-tmp\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.099605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.099547    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-config\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.099605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.099574    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-oauth-serving-cert\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.099750 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.099647    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-service-ca\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.099750 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.099666    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-oauth-config\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.099854 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.099745    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-serving-cert\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.200367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.200338    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-service-ca\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.200540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.200372    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-oauth-config\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.200540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.200394    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-serving-cert\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.200540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.200424    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-trusted-ca-bundle\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.200540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.200442    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97ngj\" (UniqueName: \"kubernetes.io/projected/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-kube-api-access-97ngj\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.200540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.200465    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-tmp\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.200540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.200498    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-config\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.200895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.200545    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-oauth-serving-cert\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.201008 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.200984    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-tmp\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.201382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.201357    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-service-ca\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.201467 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.201388    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-oauth-serving-cert\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.201537 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.201513    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-config\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.201830 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.201809    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-trusted-ca-bundle\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.203180 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.203160    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-serving-cert\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.203286 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.203248    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-oauth-config\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.214097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.214071    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97ngj\" (UniqueName: \"kubernetes.io/projected/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-kube-api-access-97ngj\") pod \"console-dfd6fc47c-vkqnx\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") " pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.378018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.377982    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:00:55.668242 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.668158    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dfd6fc47c-vkqnx"]
Oct 31 13:00:55.671826 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:00:55.671797    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa9c8797_8f1c_433b_94aa_5ab461ddfcfd.slice/crio-4a8c08b5dd66de2d67d90fa3fc35e503ac5e47d148ac7558bc128eda796dbe57 WatchSource:0}: Error finding container 4a8c08b5dd66de2d67d90fa3fc35e503ac5e47d148ac7558bc128eda796dbe57: Status 404 returned error can't find the container with id 4a8c08b5dd66de2d67d90fa3fc35e503ac5e47d148ac7558bc128eda796dbe57
Oct 31 13:00:55.930219 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.930119    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfd6fc47c-vkqnx" event={"ID":"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd","Type":"ContainerStarted","Data":"6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f"}
Oct 31 13:00:55.930219 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.930180    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfd6fc47c-vkqnx" event={"ID":"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd","Type":"ContainerStarted","Data":"4a8c08b5dd66de2d67d90fa3fc35e503ac5e47d148ac7558bc128eda796dbe57"}
Oct 31 13:00:55.952371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:00:55.952316    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-dfd6fc47c-vkqnx" podStartSLOduration=0.95230247 podStartE2EDuration="952.30247ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:00:55.951299746 +0000 UTC m=+35.367241607" watchObservedRunningTime="2026-10-31 13:00:55.95230247 +0000 UTC m=+35.368244306"
Oct 31 13:01:01.670031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:01.670004    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-b9f69b5fb-mnd5t"
Oct 31 13:01:05.378338 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:05.378298    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:01:05.378723 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:05.378379    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:01:05.383151 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:05.383129    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:01:05.976285 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:05.976254    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:01:06.050819 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:06.050775    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cd84c7c9-bhh7z"]
Oct 31 13:01:18.245942 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:01:18.245904    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:01:31.074555 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.074520    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-64cd84c7c9-bhh7z" podUID="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" containerName="console" containerID="cri-o://7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a" gracePeriod=15
Oct 31 13:01:31.317356 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.317332    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:01:31.420575 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.420490    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle\") pod \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") "
Oct 31 13:01:31.420575 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.420533    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp\") pod \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") "
Oct 31 13:01:31.420575 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.420564    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config\") pod \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") "
Oct 31 13:01:31.420888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.420609    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config\") pod \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") "
Oct 31 13:01:31.420888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.420775    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert\") pod \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") "
Oct 31 13:01:31.420888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.420824    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7\" (UniqueName: \"kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7\") pod \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") "
Oct 31 13:01:31.420888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.420851    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp" pod "1eb015cd-93dd-4a5f-a8f2-c1192a389a81" (UID: "1eb015cd-93dd-4a5f-a8f2-c1192a389a81"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:01:31.420888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.420877    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert\") pod \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") "
Oct 31 13:01:31.421142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.420955    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca\") pod \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\" (UID: \"1eb015cd-93dd-4a5f-a8f2-c1192a389a81\") "
Oct 31 13:01:31.421142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.421025    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle" pod "1eb015cd-93dd-4a5f-a8f2-c1192a389a81" (UID: "1eb015cd-93dd-4a5f-a8f2-c1192a389a81"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:31.421142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.421090    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config" pod "1eb015cd-93dd-4a5f-a8f2-c1192a389a81" (UID: "1eb015cd-93dd-4a5f-a8f2-c1192a389a81"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:31.421426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.421332    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert" pod "1eb015cd-93dd-4a5f-a8f2-c1192a389a81" (UID: "1eb015cd-93dd-4a5f-a8f2-c1192a389a81"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:31.421426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.421350    5787 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-trusted-ca-bundle\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:31.421426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.421386    5787 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-tmp\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:31.421426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.421406    5787 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-config\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:31.421622 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.421436    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca" pod "1eb015cd-93dd-4a5f-a8f2-c1192a389a81" (UID: "1eb015cd-93dd-4a5f-a8f2-c1192a389a81"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:01:31.423177 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.423156    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert" pod "1eb015cd-93dd-4a5f-a8f2-c1192a389a81" (UID: "1eb015cd-93dd-4a5f-a8f2-c1192a389a81"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:31.423268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.423192    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7" pod "1eb015cd-93dd-4a5f-a8f2-c1192a389a81" (UID: "1eb015cd-93dd-4a5f-a8f2-c1192a389a81"). InnerVolumeSpecName "kube-api-access-p7mc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:01:31.423268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.423222    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config" pod "1eb015cd-93dd-4a5f-a8f2-c1192a389a81" (UID: "1eb015cd-93dd-4a5f-a8f2-c1192a389a81"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:01:31.521896 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.521861    5787 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-service-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:31.521896 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.521898    5787 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-oauth-config\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:31.522102 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.521913    5787 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-console-serving-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:31.522102 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.521928    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7mc7\" (UniqueName: \"kubernetes.io/projected/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-kube-api-access-p7mc7\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:31.522102 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:31.521944    5787 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1eb015cd-93dd-4a5f-a8f2-c1192a389a81-oauth-serving-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:01:32.067051 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.067017    5787 generic.go:356] "Generic (PLEG): container finished" podID="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" containerID="7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a" exitCode=0
Oct 31 13:01:32.067218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.067076    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64cd84c7c9-bhh7z"
Oct 31 13:01:32.067218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.067078    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cd84c7c9-bhh7z" event={"ID":"1eb015cd-93dd-4a5f-a8f2-c1192a389a81","Type":"ContainerDied","Data":"7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a"}
Oct 31 13:01:32.067218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.067147    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64cd84c7c9-bhh7z" event={"ID":"1eb015cd-93dd-4a5f-a8f2-c1192a389a81","Type":"ContainerDied","Data":"c152106cf4fbd37551e2bdd4d53aad571ec529a904d046202a3c593c49c4b2e3"}
Oct 31 13:01:32.067218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.067169    5787 scope.go:122] "RemoveContainer" containerID="7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a"
Oct 31 13:01:32.075862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.075665    5787 scope.go:122] "RemoveContainer" containerID="7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a"
Oct 31 13:01:32.076085 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:01:32.075975    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a\": container with ID starting with 7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a not found: ID does not exist" containerID="7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a"
Oct 31 13:01:32.076085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.075998    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a"} err="failed to get container status \"7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a\": rpc error: code = NotFound desc = could not find container \"7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a\": container with ID starting with 7e00195372945d4ef4916d148793a90e71fb8678ac080abee4611f79e8c1615a not found: ID does not exist"
Oct 31 13:01:32.089379 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.089350    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64cd84c7c9-bhh7z"]
Oct 31 13:01:32.094076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.094054    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64cd84c7c9-bhh7z"]
Oct 31 13:01:32.703086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:01:32.703055    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" path="/var/lib/kubelet/pods/1eb015cd-93dd-4a5f-a8f2-c1192a389a81/volumes"
Oct 31 13:01:48.245637 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:01:48.245600    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:02:02.769640 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.769560    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9d7869469-rl9wc"]
Oct 31 13:02:02.770103 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.769885    5787 cpu_manager.go:414] "containerMap: removing container" podUID="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" containerName="console"
Oct 31 13:02:02.770103 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.769896    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" containerName="console"
Oct 31 13:02:02.770103 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.769967    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb015cd-93dd-4a5f-a8f2-c1192a389a81" containerName="console"
Oct 31 13:02:02.772760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.772744    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.785664 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.785639    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d7869469-rl9wc"]
Oct 31 13:02:02.883753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.883691    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-service-ca\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.883930 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.883762    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxglw\" (UniqueName: \"kubernetes.io/projected/de46a2dc-38e7-493c-961b-b663ae7f7c27-kube-api-access-sxglw\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.883930 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.883899    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-oauth-serving-cert\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.884001 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.883949    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de46a2dc-38e7-493c-961b-b663ae7f7c27-console-serving-cert\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.884001 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.883965    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de46a2dc-38e7-493c-961b-b663ae7f7c27-console-oauth-config\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.884075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.884055    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de46a2dc-38e7-493c-961b-b663ae7f7c27-tmp\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.884112 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.884093    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-trusted-ca-bundle\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.884145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.884123    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-console-config\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.985143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.985104    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-oauth-serving-cert\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.985328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.985154    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de46a2dc-38e7-493c-961b-b663ae7f7c27-console-serving-cert\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.985328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.985169    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de46a2dc-38e7-493c-961b-b663ae7f7c27-console-oauth-config\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.985328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.985196    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de46a2dc-38e7-493c-961b-b663ae7f7c27-tmp\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.985328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.985213    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-trusted-ca-bundle\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.985328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.985238    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-console-config\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.985328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.985255    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-service-ca\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.985328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.985281    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxglw\" (UniqueName: \"kubernetes.io/projected/de46a2dc-38e7-493c-961b-b663ae7f7c27-kube-api-access-sxglw\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.985760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.985698    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/de46a2dc-38e7-493c-961b-b663ae7f7c27-tmp\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.986205 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.986180    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-oauth-serving-cert\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.986311 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.986288    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-console-config\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.986350 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.986301    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-service-ca\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.986472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.986454    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de46a2dc-38e7-493c-961b-b663ae7f7c27-trusted-ca-bundle\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.987982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.987956    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de46a2dc-38e7-493c-961b-b663ae7f7c27-console-oauth-config\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.987982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.987969    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de46a2dc-38e7-493c-961b-b663ae7f7c27-console-serving-cert\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:02.998381 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:02.998361    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxglw\" (UniqueName: \"kubernetes.io/projected/de46a2dc-38e7-493c-961b-b663ae7f7c27-kube-api-access-sxglw\") pod \"console-9d7869469-rl9wc\" (UID: \"de46a2dc-38e7-493c-961b-b663ae7f7c27\") " pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:03.083988 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:03.083918    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:03.358672 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:03.358602    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d7869469-rl9wc"]
Oct 31 13:02:03.361739 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:02:03.361714    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde46a2dc_38e7_493c_961b_b663ae7f7c27.slice/crio-39fb737004cd5af67e8d23c5295434201bebe896f2b943fefe51b042ed4441ea WatchSource:0}: Error finding container 39fb737004cd5af67e8d23c5295434201bebe896f2b943fefe51b042ed4441ea: Status 404 returned error can't find the container with id 39fb737004cd5af67e8d23c5295434201bebe896f2b943fefe51b042ed4441ea
Oct 31 13:02:04.182310 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:04.182271    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d7869469-rl9wc" event={"ID":"de46a2dc-38e7-493c-961b-b663ae7f7c27","Type":"ContainerStarted","Data":"6f93d06e6e8a95796f09678ac70f80d8a65a80dd02ed128c848f6987fd5ee2a3"}
Oct 31 13:02:04.182762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:04.182339    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d7869469-rl9wc" event={"ID":"de46a2dc-38e7-493c-961b-b663ae7f7c27","Type":"ContainerStarted","Data":"39fb737004cd5af67e8d23c5295434201bebe896f2b943fefe51b042ed4441ea"}
Oct 31 13:02:04.206145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:04.206095    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9d7869469-rl9wc" podStartSLOduration=2.20608041 podStartE2EDuration="2.20608041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:04.205452066 +0000 UTC m=+103.621393905" watchObservedRunningTime="2026-10-31 13:02:04.20608041 +0000 UTC m=+103.622022242"
Oct 31 13:02:13.084320 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:13.084283    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:13.084733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:13.084339    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:13.088980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:13.088956    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:13.219077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:13.219044    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9d7869469-rl9wc"
Oct 31 13:02:13.273883 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:13.273851    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dfd6fc47c-vkqnx"]
Oct 31 13:02:18.245612 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:02:18.245573    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:02:38.293881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.293824    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-dfd6fc47c-vkqnx" podUID="aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" containerName="console" containerID="cri-o://6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f" gracePeriod=15
Oct 31 13:02:38.526716 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.526673    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:02:38.584452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584372    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-serving-cert\") pod \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") "
Oct 31 13:02:38.584452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584417    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-tmp\") pod \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") "
Oct 31 13:02:38.584452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584443    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-oauth-serving-cert\") pod \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") "
Oct 31 13:02:38.584699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584468    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-kube-api-access-97ngj\" (UniqueName: \"kubernetes.io/projected/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-kube-api-access-97ngj\") pod \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") "
Oct 31 13:02:38.584699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584504    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-service-ca\") pod \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") "
Oct 31 13:02:38.584699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584537    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-config\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-config\") pod \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") "
Oct 31 13:02:38.584699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584635    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-oauth-config\") pod \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") "
Oct 31 13:02:38.584949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584746    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-trusted-ca-bundle\") pod \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\" (UID: \"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd\") "
Oct 31 13:02:38.585006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584947    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-config" pod "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" (UID: "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:02:38.585006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584952    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-oauth-serving-cert" pod "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" (UID: "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:02:38.585006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.584976    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-service-ca" pod "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" (UID: "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:02:38.585250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.585166    5787 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-oauth-serving-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:38.585250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.585194    5787 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-service-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:38.585250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.585204    5787 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-config\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:38.585250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.585207    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-tmp" pod "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" (UID: "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:02:38.585639 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.585435    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-trusted-ca-bundle" pod "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" (UID: "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:02:38.586828 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.586807    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-kube-api-access-97ngj" pod "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" (UID: "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd"). InnerVolumeSpecName "kube-api-access-97ngj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:02:38.586915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.586838    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-oauth-config" pod "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" (UID: "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:02:38.586915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.586849    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-serving-cert" pod "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" (UID: "aa9c8797-8f1c-433b-94aa-5ab461ddfcfd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:02:38.685682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.685640    5787 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-trusted-ca-bundle\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:38.685682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.685678    5787 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-serving-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:38.685682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.685692    5787 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-tmp\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:38.685963 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.685722    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97ngj\" (UniqueName: \"kubernetes.io/projected/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-kube-api-access-97ngj\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:38.685963 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:38.685739    5787 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd-console-oauth-config\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:02:39.310677 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:39.310641    5787 generic.go:356] "Generic (PLEG): container finished" podID="aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" containerID="6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f" exitCode=0
Oct 31 13:02:39.311173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:39.310686    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dfd6fc47c-vkqnx"
Oct 31 13:02:39.311173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:39.310696    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfd6fc47c-vkqnx" event={"ID":"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd","Type":"ContainerDied","Data":"6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f"}
Oct 31 13:02:39.311173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:39.310771    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dfd6fc47c-vkqnx" event={"ID":"aa9c8797-8f1c-433b-94aa-5ab461ddfcfd","Type":"ContainerDied","Data":"4a8c08b5dd66de2d67d90fa3fc35e503ac5e47d148ac7558bc128eda796dbe57"}
Oct 31 13:02:39.311173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:39.310802    5787 scope.go:122] "RemoveContainer" containerID="6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f"
Oct 31 13:02:39.318737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:39.318698    5787 scope.go:122] "RemoveContainer" containerID="6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f"
Oct 31 13:02:39.319012 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:02:39.318991    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f\": container with ID starting with 6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f not found: ID does not exist" containerID="6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f"
Oct 31 13:02:39.319077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:39.319023    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f"} err="failed to get container status \"6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f\": rpc error: code = NotFound desc = could not find container \"6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f\": container with ID starting with 6a54eb2ba83a018e5d80713ab77fe98b9dc1c99caf20966888d3b02d6eb6e18f not found: ID does not exist"
Oct 31 13:02:39.330572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:39.330548    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dfd6fc47c-vkqnx"]
Oct 31 13:02:39.337071 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:39.337049    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dfd6fc47c-vkqnx"]
Oct 31 13:02:40.702947 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:02:40.702917    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" path="/var/lib/kubelet/pods/aa9c8797-8f1c-433b-94aa-5ab461ddfcfd/volumes"
Oct 31 13:02:48.245251 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:02:48.245207    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:03:18.245378 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:03:18.245340    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:03:48.245400 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:03:48.245315    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:04:18.244207 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:04:18.244168    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:04:34.486492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.486459    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"]
Oct 31 13:04:34.486880 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.486822    5787 cpu_manager.go:414] "containerMap: removing container" podUID="aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" containerName="console"
Oct 31 13:04:34.486880 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.486835    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" containerName="console"
Oct 31 13:04:34.486966 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.486891    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9c8797-8f1c-433b-94aa-5ab461ddfcfd" containerName="console"
Oct 31 13:04:34.489800 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.489782    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.499649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.499632    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 13:04:34.499768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.499697    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 13:04:34.500035 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.500021    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 13:04:34.500123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.500112    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 13:04:34.500161 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.500132    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 13:04:34.500418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.500406    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kc68z\""
Oct 31 13:04:34.502731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.502695    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 13:04:34.507656 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.507627    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r99m\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-kube-api-access-4r99m\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.507815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.507682    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.507815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.507783    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.507926 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.507829    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/060ed794-7218-4f6f-af43-9c1aabd2de62-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.507926 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.507903    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.508017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.507951    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.508017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.508008    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.508344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.508321    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"]
Oct 31 13:04:34.608987 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.608949    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.609160 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.609017    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.609160 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.609079    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r99m\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-kube-api-access-4r99m\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.609160 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.609114    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.609160 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.609138    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.609388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.609167    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/060ed794-7218-4f6f-af43-9c1aabd2de62-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.609388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.609193    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.609659 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.609628    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.611629 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.611607    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.611740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.611615    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.611919 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.611901    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/060ed794-7218-4f6f-af43-9c1aabd2de62-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.612078 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.612056    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.622123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.622097    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r99m\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-kube-api-access-4r99m\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.622856 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.622842    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-jm69m\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:34.801259 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:34.801162    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:35.081153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:35.081079    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"]
Oct 31 13:04:35.084350 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:04:35.084326    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060ed794_7218_4f6f_af43_9c1aabd2de62.slice/crio-45178ca86a37038396239a86ba2c40299dc374539fe8116008c3517224faf9d1 WatchSource:0}: Error finding container 45178ca86a37038396239a86ba2c40299dc374539fe8116008c3517224faf9d1: Status 404 returned error can't find the container with id 45178ca86a37038396239a86ba2c40299dc374539fe8116008c3517224faf9d1
Oct 31 13:04:35.727630 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:35.727600    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m" event={"ID":"060ed794-7218-4f6f-af43-9c1aabd2de62","Type":"ContainerStarted","Data":"45178ca86a37038396239a86ba2c40299dc374539fe8116008c3517224faf9d1"}
Oct 31 13:04:38.210514 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:38.210486    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:04:38.210899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:38.210558    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:04:38.742994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:38.742957    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m" event={"ID":"060ed794-7218-4f6f-af43-9c1aabd2de62","Type":"ContainerStarted","Data":"491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e"}
Oct 31 13:04:38.743201 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:38.743060    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:38.770201 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:38.770145    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m" podStartSLOduration=1.64669819 podStartE2EDuration="4.770131221s" totalImagesPullingTime="3.123433031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:04:38.768508328 +0000 UTC m=+258.184450169" watchObservedRunningTime="2026-10-31 13:04:38.770131221 +0000 UTC m=+258.186073061"
Oct 31 13:04:39.748928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:04:39.748897    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:04:48.244724 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:04:48.244670    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:05:10.989112 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:10.989030    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"]
Oct 31 13:05:10.992269 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:10.992254    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.040590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.040551    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"]
Oct 31 13:05:11.041122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.041094    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8684b1b-03a2-4f50-946a-704fccf89e3e-metrics-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.041677 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.041646    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c8684b1b-03a2-4f50-946a-704fccf89e3e-manifests\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.041836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.041764    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8684b1b-03a2-4f50-946a-704fccf89e3e-tmp\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.041836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.041793    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8684b1b-03a2-4f50-946a-704fccf89e3e-webhook-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.041939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.041852    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c8684b1b-03a2-4f50-946a-704fccf89e3e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.041939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.041902    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6rft\" (UniqueName: \"kubernetes.io/projected/c8684b1b-03a2-4f50-946a-704fccf89e3e-kube-api-access-p6rft\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.055445 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.055421    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 13:05:11.055602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.055547    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 13:05:11.055676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.055620    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 13:05:11.055795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.055782    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 13:05:11.055920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.055905    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9qk9g\""
Oct 31 13:05:11.056105 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.056087    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 13:05:11.142286 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.142252    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c8684b1b-03a2-4f50-946a-704fccf89e3e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.142468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.142295    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6rft\" (UniqueName: \"kubernetes.io/projected/c8684b1b-03a2-4f50-946a-704fccf89e3e-kube-api-access-p6rft\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.142468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.142330    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8684b1b-03a2-4f50-946a-704fccf89e3e-metrics-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.142468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.142346    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c8684b1b-03a2-4f50-946a-704fccf89e3e-manifests\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.142468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.142385    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8684b1b-03a2-4f50-946a-704fccf89e3e-tmp\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.142468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.142403    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8684b1b-03a2-4f50-946a-704fccf89e3e-webhook-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.142909 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.142887    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c8684b1b-03a2-4f50-946a-704fccf89e3e-manifests\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.144805 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.144773    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8684b1b-03a2-4f50-946a-704fccf89e3e-tmp\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.145000 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.144983    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c8684b1b-03a2-4f50-946a-704fccf89e3e-metrics-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.145069 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.145017    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c8684b1b-03a2-4f50-946a-704fccf89e3e-webhook-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.145124 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.145066    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c8684b1b-03a2-4f50-946a-704fccf89e3e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.157113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.157093    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6rft\" (UniqueName: \"kubernetes.io/projected/c8684b1b-03a2-4f50-946a-704fccf89e3e-kube-api-access-p6rft\") pod \"opendatahub-operator-controller-manager-849556f89d-hf7p8\" (UID: \"c8684b1b-03a2-4f50-946a-704fccf89e3e\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.306851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.306771    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:11.672540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.672509    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"]
Oct 31 13:05:11.672933 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:05:11.672901    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8684b1b_03a2_4f50_946a_704fccf89e3e.slice/crio-95a32cdfe790a758584ade3cac1fed07e2f96c5e373ee91dae26152f1018ecb6 WatchSource:0}: Error finding container 95a32cdfe790a758584ade3cac1fed07e2f96c5e373ee91dae26152f1018ecb6: Status 404 returned error can't find the container with id 95a32cdfe790a758584ade3cac1fed07e2f96c5e373ee91dae26152f1018ecb6
Oct 31 13:05:11.868542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:11.868502    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8" event={"ID":"c8684b1b-03a2-4f50-946a-704fccf89e3e","Type":"ContainerStarted","Data":"95a32cdfe790a758584ade3cac1fed07e2f96c5e373ee91dae26152f1018ecb6"}
Oct 31 13:05:15.889177 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:15.889090    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8" event={"ID":"c8684b1b-03a2-4f50-946a-704fccf89e3e","Type":"ContainerStarted","Data":"086d57fb3d31ea5603d35b3dc0f23702c55dc3aae470d45f43709873b8d607ca"}
Oct 31 13:05:16.894954 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:16.894920    5787 generic.go:356] "Generic (PLEG): container finished" podID="c8684b1b-03a2-4f50-946a-704fccf89e3e" containerID="086d57fb3d31ea5603d35b3dc0f23702c55dc3aae470d45f43709873b8d607ca" exitCode=0
Oct 31 13:05:16.895362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:16.894995    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8" event={"ID":"c8684b1b-03a2-4f50-946a-704fccf89e3e","Type":"ContainerDied","Data":"086d57fb3d31ea5603d35b3dc0f23702c55dc3aae470d45f43709873b8d607ca"}
Oct 31 13:05:17.901374 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:17.901339    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8" event={"ID":"c8684b1b-03a2-4f50-946a-704fccf89e3e","Type":"ContainerStarted","Data":"fb89b9757e3e04040dbe99d8266e44ebcd825dd9497fb252ad5d6931033b84fb"}
Oct 31 13:05:17.901776 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:17.901412    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:17.948602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:17.948545    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8" podStartSLOduration=3.332407402 podStartE2EDuration="7.948531485s" totalImagesPullingTime="4.178632463s" totalInitContainerRuntime="437.49162ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:17.946506465 +0000 UTC m=+297.362448329" watchObservedRunningTime="2026-10-31 13:05:17.948531485 +0000 UTC m=+297.364473325"
Oct 31 13:05:18.246210 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:05:18.246166    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:05:28.908549 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:28.908514    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-849556f89d-hf7p8"
Oct 31 13:05:40.037660 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.037621    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"]
Oct 31 13:05:40.040920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.040903    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"
Oct 31 13:05:40.047346 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.047327    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 13:05:40.047346 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.047327    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dt4bx\""
Oct 31 13:05:40.047546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.047326    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 13:05:40.055115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.055094    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"]
Oct 31 13:05:40.086648 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.086613    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rt8\" (UniqueName: \"kubernetes.io/projected/e5964761-efa2-4a50-82c8-ae9a984603a9-kube-api-access-n5rt8\") pod \"authorino-operator-7b7bb4bc77-gc6kh\" (UID: \"e5964761-efa2-4a50-82c8-ae9a984603a9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"
Oct 31 13:05:40.187292 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.187262    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rt8\" (UniqueName: \"kubernetes.io/projected/e5964761-efa2-4a50-82c8-ae9a984603a9-kube-api-access-n5rt8\") pod \"authorino-operator-7b7bb4bc77-gc6kh\" (UID: \"e5964761-efa2-4a50-82c8-ae9a984603a9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"
Oct 31 13:05:40.205587 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.205567    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rt8\" (UniqueName: \"kubernetes.io/projected/e5964761-efa2-4a50-82c8-ae9a984603a9-kube-api-access-n5rt8\") pod \"authorino-operator-7b7bb4bc77-gc6kh\" (UID: \"e5964761-efa2-4a50-82c8-ae9a984603a9\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"
Oct 31 13:05:40.353605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.353522    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"
Oct 31 13:05:40.630281 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.630207    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"]
Oct 31 13:05:40.633166 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:05:40.633134    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5964761_efa2_4a50_82c8_ae9a984603a9.slice/crio-afae8cec31372fc1eb19cf1bb81bd8a15bcf4a3f19bec2b7d20ebbc426ac2899 WatchSource:0}: Error finding container afae8cec31372fc1eb19cf1bb81bd8a15bcf4a3f19bec2b7d20ebbc426ac2899: Status 404 returned error can't find the container with id afae8cec31372fc1eb19cf1bb81bd8a15bcf4a3f19bec2b7d20ebbc426ac2899
Oct 31 13:05:40.995029 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:40.994996    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh" event={"ID":"e5964761-efa2-4a50-82c8-ae9a984603a9","Type":"ContainerStarted","Data":"afae8cec31372fc1eb19cf1bb81bd8a15bcf4a3f19bec2b7d20ebbc426ac2899"}
Oct 31 13:05:41.656409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.656374    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"]
Oct 31 13:05:41.661053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.661037    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.668324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.668305    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 31 13:05:41.668548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.668527    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9kb5b\""
Oct 31 13:05:41.671080 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.671066    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 31 13:05:41.679979 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.679960    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"]
Oct 31 13:05:41.800239 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.800206    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn6sg\" (UniqueName: \"kubernetes.io/projected/4d80a741-9910-40ee-90a7-7ef3a71d1c55-kube-api-access-zn6sg\") pod \"kuadrant-console-plugin-66c4fb575d-pfgg8\" (UID: \"4d80a741-9910-40ee-90a7-7ef3a71d1c55\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.800398 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.800313    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4d80a741-9910-40ee-90a7-7ef3a71d1c55-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pfgg8\" (UID: \"4d80a741-9910-40ee-90a7-7ef3a71d1c55\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.800398 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.800341    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d80a741-9910-40ee-90a7-7ef3a71d1c55-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pfgg8\" (UID: \"4d80a741-9910-40ee-90a7-7ef3a71d1c55\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.900849 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.900810    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn6sg\" (UniqueName: \"kubernetes.io/projected/4d80a741-9910-40ee-90a7-7ef3a71d1c55-kube-api-access-zn6sg\") pod \"kuadrant-console-plugin-66c4fb575d-pfgg8\" (UID: \"4d80a741-9910-40ee-90a7-7ef3a71d1c55\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.901097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.900888    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4d80a741-9910-40ee-90a7-7ef3a71d1c55-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pfgg8\" (UID: \"4d80a741-9910-40ee-90a7-7ef3a71d1c55\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.901097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.900927    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d80a741-9910-40ee-90a7-7ef3a71d1c55-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pfgg8\" (UID: \"4d80a741-9910-40ee-90a7-7ef3a71d1c55\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.901785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.901679    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4d80a741-9910-40ee-90a7-7ef3a71d1c55-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-pfgg8\" (UID: \"4d80a741-9910-40ee-90a7-7ef3a71d1c55\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.903872 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.903846    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d80a741-9910-40ee-90a7-7ef3a71d1c55-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-pfgg8\" (UID: \"4d80a741-9910-40ee-90a7-7ef3a71d1c55\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.915254 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.915193    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn6sg\" (UniqueName: \"kubernetes.io/projected/4d80a741-9910-40ee-90a7-7ef3a71d1c55-kube-api-access-zn6sg\") pod \"kuadrant-console-plugin-66c4fb575d-pfgg8\" (UID: \"4d80a741-9910-40ee-90a7-7ef3a71d1c55\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:41.985649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:41.985614    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"
Oct 31 13:05:42.269042 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:42.269013    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8"]
Oct 31 13:05:42.272410 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:05:42.272385    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d80a741_9910_40ee_90a7_7ef3a71d1c55.slice/crio-5464c3eaebfeec5ec462111852a3fbb8aca871301734e2b8c3ca3e4ea37fa30d WatchSource:0}: Error finding container 5464c3eaebfeec5ec462111852a3fbb8aca871301734e2b8c3ca3e4ea37fa30d: Status 404 returned error can't find the container with id 5464c3eaebfeec5ec462111852a3fbb8aca871301734e2b8c3ca3e4ea37fa30d
Oct 31 13:05:43.006030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:43.005990    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8" event={"ID":"4d80a741-9910-40ee-90a7-7ef3a71d1c55","Type":"ContainerStarted","Data":"5464c3eaebfeec5ec462111852a3fbb8aca871301734e2b8c3ca3e4ea37fa30d"}
Oct 31 13:05:45.016677 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:45.016639    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh" event={"ID":"e5964761-efa2-4a50-82c8-ae9a984603a9","Type":"ContainerStarted","Data":"79582dc064a4039606ec2241cf0098acda817682b6b67e42a06e2db7d30021bd"}
Oct 31 13:05:45.017141 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:45.016918    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"
Oct 31 13:05:45.042040 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:45.041987    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh" podStartSLOduration=1.1643820009999999 podStartE2EDuration="5.041973925s" totalImagesPullingTime="3.877591924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:45.041185738 +0000 UTC m=+324.457127580" watchObservedRunningTime="2026-10-31 13:05:45.041973925 +0000 UTC m=+324.457915760"
Oct 31 13:05:48.253683 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:05:48.253643    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:05:49.034914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:49.034875    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8" event={"ID":"4d80a741-9910-40ee-90a7-7ef3a71d1c55","Type":"ContainerStarted","Data":"cb93d14b35557000c2ed08a634e3f0425d8b053828f0310cfbae01b2a725760d"}
Oct 31 13:05:56.022862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:56.022834    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gc6kh"
Oct 31 13:05:56.044867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:05:56.044822    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-pfgg8" podStartSLOduration=8.76954636 podStartE2EDuration="15.044806452s" totalImagesPullingTime="6.275260092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:49.069266789 +0000 UTC m=+328.485208629" watchObservedRunningTime="2026-10-31 13:05:56.044806452 +0000 UTC m=+335.460748292"
Oct 31 13:06:06.334886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.334848    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"]
Oct 31 13:06:06.338115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.338098    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.344542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.344424    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 13:06:06.344542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.344427    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-x4h6t\""
Oct 31 13:06:06.344780 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.344761    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 13:06:06.344851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.344815    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 13:06:06.345340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.345308    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 13:06:06.345442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.345397    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 13:06:06.416154 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.416116    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"]
Oct 31 13:06:06.432404 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.432373    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxhm\" (UniqueName: \"kubernetes.io/projected/ad206c3e-f20a-48bf-9290-7c681aaddfac-kube-api-access-hzxhm\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.432548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.432421    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad206c3e-f20a-48bf-9290-7c681aaddfac-cert\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.432627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.432602    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad206c3e-f20a-48bf-9290-7c681aaddfac-metrics-certs\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.432671 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.432643    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad206c3e-f20a-48bf-9290-7c681aaddfac-manager-config\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.533757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.533718    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad206c3e-f20a-48bf-9290-7c681aaddfac-metrics-certs\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.533943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.533762    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad206c3e-f20a-48bf-9290-7c681aaddfac-manager-config\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.533943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.533825    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxhm\" (UniqueName: \"kubernetes.io/projected/ad206c3e-f20a-48bf-9290-7c681aaddfac-kube-api-access-hzxhm\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.533943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.533875    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad206c3e-f20a-48bf-9290-7c681aaddfac-cert\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.534516 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.534494    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/ad206c3e-f20a-48bf-9290-7c681aaddfac-manager-config\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.536300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.536277    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ad206c3e-f20a-48bf-9290-7c681aaddfac-cert\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.536383 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.536350    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad206c3e-f20a-48bf-9290-7c681aaddfac-metrics-certs\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.549216 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.549193    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxhm\" (UniqueName: \"kubernetes.io/projected/ad206c3e-f20a-48bf-9290-7c681aaddfac-kube-api-access-hzxhm\") pod \"jobset-controller-manager-5c748f66b6-g58tt\" (UID: \"ad206c3e-f20a-48bf-9290-7c681aaddfac\") " pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.650481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.650388    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:06.924006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.923924    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"]
Oct 31 13:06:06.926879 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:06:06.926857    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad206c3e_f20a_48bf_9290_7c681aaddfac.slice/crio-ddc33911db2a5d884286ef770d77ed6d994b5c6f8e40c86436e1eb01693ea779 WatchSource:0}: Error finding container ddc33911db2a5d884286ef770d77ed6d994b5c6f8e40c86436e1eb01693ea779: Status 404 returned error can't find the container with id ddc33911db2a5d884286ef770d77ed6d994b5c6f8e40c86436e1eb01693ea779
Oct 31 13:06:06.928749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:06.928733    5787 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:06:07.109441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:07.109411    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt" event={"ID":"ad206c3e-f20a-48bf-9290-7c681aaddfac","Type":"ContainerStarted","Data":"ddc33911db2a5d884286ef770d77ed6d994b5c6f8e40c86436e1eb01693ea779"}
Oct 31 13:06:11.132363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:11.132268    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt" event={"ID":"ad206c3e-f20a-48bf-9290-7c681aaddfac","Type":"ContainerStarted","Data":"9e2bd6a544ae9517bdf1466b8ccce97c7840074ed365d3ba39da8e43acd05ae6"}
Oct 31 13:06:11.132845 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:11.132424    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:11.153848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:11.153785    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt" podStartSLOduration=1.233279886 podStartE2EDuration="5.153769792s" totalImagesPullingTime="3.920489906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:11.153290733 +0000 UTC m=+350.569232574" watchObservedRunningTime="2026-10-31 13:06:11.153769792 +0000 UTC m=+350.569711630"
Oct 31 13:06:18.246734 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:06:18.246675    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:06:22.143267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:22.143234    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-5c748f66b6-g58tt"
Oct 31 13:06:26.974821 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:26.974783    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lkmx4"]
Oct 31 13:06:26.999311 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:26.999282    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lkmx4"]
Oct 31 13:06:26.999466 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:26.999377    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4"
Oct 31 13:06:27.008345 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:27.008319    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ddb8z\""
Oct 31 13:06:27.124869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:27.124822    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh8bt\" (UniqueName: \"kubernetes.io/projected/caac3829-d20c-45c1-ac3f-33b36f5afd6a-kube-api-access-rh8bt\") pod \"authorino-56c6c9ddd5-lkmx4\" (UID: \"caac3829-d20c-45c1-ac3f-33b36f5afd6a\") " pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4"
Oct 31 13:06:27.225407 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:27.225325    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh8bt\" (UniqueName: \"kubernetes.io/projected/caac3829-d20c-45c1-ac3f-33b36f5afd6a-kube-api-access-rh8bt\") pod \"authorino-56c6c9ddd5-lkmx4\" (UID: \"caac3829-d20c-45c1-ac3f-33b36f5afd6a\") " pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4"
Oct 31 13:06:27.241582 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:27.241553    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh8bt\" (UniqueName: \"kubernetes.io/projected/caac3829-d20c-45c1-ac3f-33b36f5afd6a-kube-api-access-rh8bt\") pod \"authorino-56c6c9ddd5-lkmx4\" (UID: \"caac3829-d20c-45c1-ac3f-33b36f5afd6a\") " pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4"
Oct 31 13:06:27.311510 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:27.311481    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4"
Oct 31 13:06:27.582912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:27.582827    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lkmx4"]
Oct 31 13:06:27.585841 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:06:27.585813    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaac3829_d20c_45c1_ac3f_33b36f5afd6a.slice/crio-a31dab26a9b72683dcfcdfb1985a34612409c9fb687e6c46694c0fa301a06cb5 WatchSource:0}: Error finding container a31dab26a9b72683dcfcdfb1985a34612409c9fb687e6c46694c0fa301a06cb5: Status 404 returned error can't find the container with id a31dab26a9b72683dcfcdfb1985a34612409c9fb687e6c46694c0fa301a06cb5
Oct 31 13:06:28.222699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:28.222663    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4" event={"ID":"caac3829-d20c-45c1-ac3f-33b36f5afd6a","Type":"ContainerStarted","Data":"a31dab26a9b72683dcfcdfb1985a34612409c9fb687e6c46694c0fa301a06cb5"}
Oct 31 13:06:30.233794 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:30.233749    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4" event={"ID":"caac3829-d20c-45c1-ac3f-33b36f5afd6a","Type":"ContainerStarted","Data":"f0041c915d6fdc7cdb2aabe1c357a34ea4bcd08343aa868c87714e2c282cc94f"}
Oct 31 13:06:30.254861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:30.254797    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4" podStartSLOduration=2.512619943 podStartE2EDuration="4.254776955s" totalImagesPullingTime="1.742157012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:30.25365292 +0000 UTC m=+369.669594761" watchObservedRunningTime="2026-10-31 13:06:30.254776955 +0000 UTC m=+369.670718795"
Oct 31 13:06:35.122001 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.121972    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9klng"]
Oct 31 13:06:35.125370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.125350    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:06:35.134352 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.134322    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wt6tq\""
Oct 31 13:06:35.134467 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.134428    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 13:06:35.134467 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.134434    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 13:06:35.134548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.134466    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 13:06:35.147311 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.147287    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"]
Oct 31 13:06:35.150772 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.150747    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9klng"]
Oct 31 13:06:35.150899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.150839    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:06:35.159623 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.159566    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 31 13:06:35.163157 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.163129    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"]
Oct 31 13:06:35.168651 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.168631    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6w67n\""
Oct 31 13:06:35.302551 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.302521    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l2rhz\" (UID: \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:06:35.302551 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.302555    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg85k\" (UniqueName: \"kubernetes.io/projected/dc065dda-a2b2-4b71-a8f8-8cc9298e40b7-kube-api-access-lg85k\") pod \"kserve-controller-manager-776cc66d47-9klng\" (UID: \"dc065dda-a2b2-4b71-a8f8-8cc9298e40b7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:06:35.302759 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.302593    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc065dda-a2b2-4b71-a8f8-8cc9298e40b7-cert\") pod \"kserve-controller-manager-776cc66d47-9klng\" (UID: \"dc065dda-a2b2-4b71-a8f8-8cc9298e40b7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:06:35.302800 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.302756    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpx9b\" (UniqueName: \"kubernetes.io/projected/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-kube-api-access-kpx9b\") pod \"llmisvc-controller-manager-869d5dcc5d-l2rhz\" (UID: \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:06:35.403926 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.403853    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc065dda-a2b2-4b71-a8f8-8cc9298e40b7-cert\") pod \"kserve-controller-manager-776cc66d47-9klng\" (UID: \"dc065dda-a2b2-4b71-a8f8-8cc9298e40b7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:06:35.403926 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.403913    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpx9b\" (UniqueName: \"kubernetes.io/projected/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-kube-api-access-kpx9b\") pod \"llmisvc-controller-manager-869d5dcc5d-l2rhz\" (UID: \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:06:35.404110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.403953    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l2rhz\" (UID: \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:06:35.404110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.403976    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg85k\" (UniqueName: \"kubernetes.io/projected/dc065dda-a2b2-4b71-a8f8-8cc9298e40b7-kube-api-access-lg85k\") pod \"kserve-controller-manager-776cc66d47-9klng\" (UID: \"dc065dda-a2b2-4b71-a8f8-8cc9298e40b7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:06:35.406582 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.406553    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dc065dda-a2b2-4b71-a8f8-8cc9298e40b7-cert\") pod \"kserve-controller-manager-776cc66d47-9klng\" (UID: \"dc065dda-a2b2-4b71-a8f8-8cc9298e40b7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:06:35.406700 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.406553    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-l2rhz\" (UID: \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:06:35.417581 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.417554    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpx9b\" (UniqueName: \"kubernetes.io/projected/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-kube-api-access-kpx9b\") pod \"llmisvc-controller-manager-869d5dcc5d-l2rhz\" (UID: \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:06:35.421440 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.421416    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg85k\" (UniqueName: \"kubernetes.io/projected/dc065dda-a2b2-4b71-a8f8-8cc9298e40b7-kube-api-access-lg85k\") pod \"kserve-controller-manager-776cc66d47-9klng\" (UID: \"dc065dda-a2b2-4b71-a8f8-8cc9298e40b7\") " pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:06:35.439350 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.439322    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:06:35.466474 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.466450    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:06:35.533765 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.533350    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"]
Oct 31 13:06:35.537836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.537813    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.564562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.564530    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"]
Oct 31 13:06:35.706515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.706490    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.706515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.706520    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d192d1e3-c83f-4e9e-979f-684a6b98cc73-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.706745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.706540    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d192d1e3-c83f-4e9e-979f-684a6b98cc73-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.706745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.706618    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5gsg\" (UniqueName: \"kubernetes.io/projected/d192d1e3-c83f-4e9e-979f-684a6b98cc73-kube-api-access-m5gsg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.706815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.706775    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.706815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.706807    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.706888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.706835    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.718083 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.718056    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-9klng"]
Oct 31 13:06:35.719843 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:06:35.719819    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc065dda_a2b2_4b71_a8f8_8cc9298e40b7.slice/crio-2544f7a0aa753c697795888473bc55c0de95ad871d59a2408c411e93aa419137 WatchSource:0}: Error finding container 2544f7a0aa753c697795888473bc55c0de95ad871d59a2408c411e93aa419137: Status 404 returned error can't find the container with id 2544f7a0aa753c697795888473bc55c0de95ad871d59a2408c411e93aa419137
Oct 31 13:06:35.757663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.757643    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"]
Oct 31 13:06:35.760878 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:06:35.760845    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf6eef8c9_3f0a_4560_b20d_7bf495c771b8.slice/crio-f49bdc60f36842aec82f38108e251875f0850733e674fed66330c1b459b33acd WatchSource:0}: Error finding container f49bdc60f36842aec82f38108e251875f0850733e674fed66330c1b459b33acd: Status 404 returned error can't find the container with id f49bdc60f36842aec82f38108e251875f0850733e674fed66330c1b459b33acd
Oct 31 13:06:35.807453 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.807423    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.807611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.807457    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d192d1e3-c83f-4e9e-979f-684a6b98cc73-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.807611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.807480    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d192d1e3-c83f-4e9e-979f-684a6b98cc73-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.807611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.807526    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5gsg\" (UniqueName: \"kubernetes.io/projected/d192d1e3-c83f-4e9e-979f-684a6b98cc73-kube-api-access-m5gsg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.807611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.807593    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.807863 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.807621    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.807863 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.807667    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.808470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.808441    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.810581 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.810550    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d192d1e3-c83f-4e9e-979f-684a6b98cc73-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.810672 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.810607    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.810736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.810667    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d192d1e3-c83f-4e9e-979f-684a6b98cc73-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.810778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.810734    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.820872 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.820849    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d192d1e3-c83f-4e9e-979f-684a6b98cc73-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.821347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.821331    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5gsg\" (UniqueName: \"kubernetes.io/projected/d192d1e3-c83f-4e9e-979f-684a6b98cc73-kube-api-access-m5gsg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-9bt9l\" (UID: \"d192d1e3-c83f-4e9e-979f-684a6b98cc73\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:35.850071 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:35.850054    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:36.171389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:36.171362    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"]
Oct 31 13:06:36.174114 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:06:36.174079    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd192d1e3_c83f_4e9e_979f_684a6b98cc73.slice/crio-73792306d3a2f09f98f6860e01068be82c7c6e28f8e12cd13d42bdf9dec02f15 WatchSource:0}: Error finding container 73792306d3a2f09f98f6860e01068be82c7c6e28f8e12cd13d42bdf9dec02f15: Status 404 returned error can't find the container with id 73792306d3a2f09f98f6860e01068be82c7c6e28f8e12cd13d42bdf9dec02f15
Oct 31 13:06:36.176373 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:36.176338    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:06:36.176463 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:36.176415    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:06:36.264362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:36.263387    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l" event={"ID":"d192d1e3-c83f-4e9e-979f-684a6b98cc73","Type":"ContainerStarted","Data":"39898dd96cec36be58bc97b036ee84971a1e257b96c77a779aafcc87943753eb"}
Oct 31 13:06:36.264362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:36.263449    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l" event={"ID":"d192d1e3-c83f-4e9e-979f-684a6b98cc73","Type":"ContainerStarted","Data":"73792306d3a2f09f98f6860e01068be82c7c6e28f8e12cd13d42bdf9dec02f15"}
Oct 31 13:06:36.264362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:36.263951    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:36.265353 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:36.265316    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz" event={"ID":"f6eef8c9-3f0a-4560-b20d-7bf495c771b8","Type":"ContainerStarted","Data":"f49bdc60f36842aec82f38108e251875f0850733e674fed66330c1b459b33acd"}
Oct 31 13:06:36.267058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:36.267031    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9klng" event={"ID":"dc065dda-a2b2-4b71-a8f8-8cc9298e40b7","Type":"ContainerStarted","Data":"2544f7a0aa753c697795888473bc55c0de95ad871d59a2408c411e93aa419137"}
Oct 31 13:06:36.295981 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:36.295917    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l" podStartSLOduration=1.295897264 podStartE2EDuration="1.295897264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:36.292360658 +0000 UTC m=+375.708302500" watchObservedRunningTime="2026-10-31 13:06:36.295897264 +0000 UTC m=+375.711839106"
Oct 31 13:06:37.276393 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.276165    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-9bt9l"
Oct 31 13:06:37.365801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.365766    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"]
Oct 31 13:06:37.366357 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.366297    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m" podUID="060ed794-7218-4f6f-af43-9c1aabd2de62" containerName="discovery" containerID="cri-o://491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e" gracePeriod=30
Oct 31 13:06:37.656916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.656891    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:06:37.833862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.833775    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-cacerts\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-cacerts\") pod \"060ed794-7218-4f6f-af43-9c1aabd2de62\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") "
Oct 31 13:06:37.833862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.833845    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-token\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-token\") pod \"060ed794-7218-4f6f-af43-9c1aabd2de62\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") "
Oct 31 13:06:37.834063 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.833877    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-ca-configmap\") pod \"060ed794-7218-4f6f-af43-9c1aabd2de62\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") "
Oct 31 13:06:37.834063 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.833942    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060ed794-7218-4f6f-af43-9c1aabd2de62-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/060ed794-7218-4f6f-af43-9c1aabd2de62-local-certs\") pod \"060ed794-7218-4f6f-af43-9c1aabd2de62\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") "
Oct 31 13:06:37.834063 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.833969    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-kubeconfig\") pod \"060ed794-7218-4f6f-af43-9c1aabd2de62\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") "
Oct 31 13:06:37.834063 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.834044    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-kube-api-access-4r99m\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-kube-api-access-4r99m\") pod \"060ed794-7218-4f6f-af43-9c1aabd2de62\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") "
Oct 31 13:06:37.834264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.834071    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-dns-cert\") pod \"060ed794-7218-4f6f-af43-9c1aabd2de62\" (UID: \"060ed794-7218-4f6f-af43-9c1aabd2de62\") "
Oct 31 13:06:37.834431 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.834378    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-ca-configmap" pod "060ed794-7218-4f6f-af43-9c1aabd2de62" (UID: "060ed794-7218-4f6f-af43-9c1aabd2de62"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:06:37.837036 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.837005    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-dns-cert" pod "060ed794-7218-4f6f-af43-9c1aabd2de62" (UID: "060ed794-7218-4f6f-af43-9c1aabd2de62"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:06:37.837421 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.837381    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060ed794-7218-4f6f-af43-9c1aabd2de62-local-certs" pod "060ed794-7218-4f6f-af43-9c1aabd2de62" (UID: "060ed794-7218-4f6f-af43-9c1aabd2de62"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:06:37.837583 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.837535    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-token" pod "060ed794-7218-4f6f-af43-9c1aabd2de62" (UID: "060ed794-7218-4f6f-af43-9c1aabd2de62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:06:37.837727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.837676    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-cacerts" pod "060ed794-7218-4f6f-af43-9c1aabd2de62" (UID: "060ed794-7218-4f6f-af43-9c1aabd2de62"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:06:37.837907 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.837850    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-kube-api-access-4r99m" pod "060ed794-7218-4f6f-af43-9c1aabd2de62" (UID: "060ed794-7218-4f6f-af43-9c1aabd2de62"). InnerVolumeSpecName "kube-api-access-4r99m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:06:37.840007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.839968    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-kubeconfig" pod "060ed794-7218-4f6f-af43-9c1aabd2de62" (UID: "060ed794-7218-4f6f-af43-9c1aabd2de62"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:06:37.935679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.935649    5787 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-cacerts\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:37.935679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.935679    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:37.935906 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.935695    5787 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-ca-configmap\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:37.935906 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.935734    5787 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/060ed794-7218-4f6f-af43-9c1aabd2de62-local-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:37.935906 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.935748    5787 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-kubeconfig\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:37.935906 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.935762    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r99m\" (UniqueName: \"kubernetes.io/projected/060ed794-7218-4f6f-af43-9c1aabd2de62-kube-api-access-4r99m\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:37.935906 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:37.935770    5787 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/060ed794-7218-4f6f-af43-9c1aabd2de62-istio-csr-dns-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:06:38.280667 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.280622    5787 generic.go:356] "Generic (PLEG): container finished" podID="060ed794-7218-4f6f-af43-9c1aabd2de62" containerID="491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e" exitCode=0
Oct 31 13:06:38.281156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.280724    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"
Oct 31 13:06:38.281156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.280726    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m" event={"ID":"060ed794-7218-4f6f-af43-9c1aabd2de62","Type":"ContainerDied","Data":"491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e"}
Oct 31 13:06:38.281156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.280895    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m" event={"ID":"060ed794-7218-4f6f-af43-9c1aabd2de62","Type":"ContainerDied","Data":"45178ca86a37038396239a86ba2c40299dc374539fe8116008c3517224faf9d1"}
Oct 31 13:06:38.281156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.280922    5787 scope.go:122] "RemoveContainer" containerID="491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e"
Oct 31 13:06:38.290544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.290527    5787 scope.go:122] "RemoveContainer" containerID="491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e"
Oct 31 13:06:38.290956 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:06:38.290931    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e\": container with ID starting with 491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e not found: ID does not exist" containerID="491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e"
Oct 31 13:06:38.291061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.290970    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e"} err="failed to get container status \"491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e\": rpc error: code = NotFound desc = could not find container \"491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e\": container with ID starting with 491c7c1be61ad298d6ae3e8e977c603ead6f0976d7c633088705880d5993454e not found: ID does not exist"
Oct 31 13:06:38.319596 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.309322    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"]
Oct 31 13:06:38.319596 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.315430    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-jm69m"]
Oct 31 13:06:38.707735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:38.705675    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060ed794-7218-4f6f-af43-9c1aabd2de62" path="/var/lib/kubelet/pods/060ed794-7218-4f6f-af43-9c1aabd2de62/volumes"
Oct 31 13:06:39.287590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:39.287559    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz" event={"ID":"f6eef8c9-3f0a-4560-b20d-7bf495c771b8","Type":"ContainerStarted","Data":"884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c"}
Oct 31 13:06:39.288101 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:39.287691    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:06:39.308895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:39.308840    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz" podStartSLOduration=1.295925312 podStartE2EDuration="4.308826967s" totalImagesPullingTime="3.012901655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:39.307405732 +0000 UTC m=+378.723347571" watchObservedRunningTime="2026-10-31 13:06:39.308826967 +0000 UTC m=+378.724768806"
Oct 31 13:06:40.293082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:40.293045    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-9klng" event={"ID":"dc065dda-a2b2-4b71-a8f8-8cc9298e40b7","Type":"ContainerStarted","Data":"61cc180d56a6d51c6e308445bb9dd84aadaac67b2d937043d87e1631e0a567ee"}
Oct 31 13:06:40.293460 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:40.293353    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:06:40.321511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:06:40.321460    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-9klng" podStartSLOduration=1.688963807 podStartE2EDuration="5.321445984s" totalImagesPullingTime="3.632482177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:40.320472383 +0000 UTC m=+379.736414231" watchObservedRunningTime="2026-10-31 13:06:40.321445984 +0000 UTC m=+379.737387823"
Oct 31 13:06:48.246033 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:06:48.245993    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:07:10.296063 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:10.296030    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:07:11.303377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:11.303346    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-9klng"
Oct 31 13:07:18.245038 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:07:18.245001    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:07:48.246082 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:07:48.246044    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:07:58.508412 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:58.508380    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lkmx4"]
Oct 31 13:07:58.508985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:58.508585    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4" podUID="caac3829-d20c-45c1-ac3f-33b36f5afd6a" containerName="authorino" containerID="cri-o://f0041c915d6fdc7cdb2aabe1c357a34ea4bcd08343aa868c87714e2c282cc94f" gracePeriod=30
Oct 31 13:07:58.640933 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:58.640897    5787 generic.go:356] "Generic (PLEG): container finished" podID="caac3829-d20c-45c1-ac3f-33b36f5afd6a" containerID="f0041c915d6fdc7cdb2aabe1c357a34ea4bcd08343aa868c87714e2c282cc94f" exitCode=0
Oct 31 13:07:58.641110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:58.640990    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4" event={"ID":"caac3829-d20c-45c1-ac3f-33b36f5afd6a","Type":"ContainerDied","Data":"f0041c915d6fdc7cdb2aabe1c357a34ea4bcd08343aa868c87714e2c282cc94f"}
Oct 31 13:07:58.752834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:58.752809    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4"
Oct 31 13:07:58.890327 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:58.890245    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caac3829-d20c-45c1-ac3f-33b36f5afd6a-kube-api-access-rh8bt\" (UniqueName: \"kubernetes.io/projected/caac3829-d20c-45c1-ac3f-33b36f5afd6a-kube-api-access-rh8bt\") pod \"caac3829-d20c-45c1-ac3f-33b36f5afd6a\" (UID: \"caac3829-d20c-45c1-ac3f-33b36f5afd6a\") "
Oct 31 13:07:58.892689 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:58.892664    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caac3829-d20c-45c1-ac3f-33b36f5afd6a-kube-api-access-rh8bt" pod "caac3829-d20c-45c1-ac3f-33b36f5afd6a" (UID: "caac3829-d20c-45c1-ac3f-33b36f5afd6a"). InnerVolumeSpecName "kube-api-access-rh8bt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:07:58.992118 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:58.992082    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh8bt\" (UniqueName: \"kubernetes.io/projected/caac3829-d20c-45c1-ac3f-33b36f5afd6a-kube-api-access-rh8bt\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:07:59.385451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.385418    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"]
Oct 31 13:07:59.385830 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.385817    5787 cpu_manager.go:414] "containerMap: removing container" podUID="060ed794-7218-4f6f-af43-9c1aabd2de62" containerName="discovery"
Oct 31 13:07:59.385876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.385831    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060ed794-7218-4f6f-af43-9c1aabd2de62" containerName="discovery"
Oct 31 13:07:59.385876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.385847    5787 cpu_manager.go:414] "containerMap: removing container" podUID="caac3829-d20c-45c1-ac3f-33b36f5afd6a" containerName="authorino"
Oct 31 13:07:59.385876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.385852    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caac3829-d20c-45c1-ac3f-33b36f5afd6a" containerName="authorino"
Oct 31 13:07:59.385972 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.385955    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="060ed794-7218-4f6f-af43-9c1aabd2de62" containerName="discovery"
Oct 31 13:07:59.385972 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.385964    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="caac3829-d20c-45c1-ac3f-33b36f5afd6a" containerName="authorino"
Oct 31 13:07:59.387946 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.387932    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:07:59.399643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.399618    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"]
Oct 31 13:07:59.496780 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.496747    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5802a4b8-1826-4452-aab5-f8d2b2bc95a1-cert\") pod \"llmisvc-controller-manager-6466c554d8-5xbhb\" (UID: \"5802a4b8-1826-4452-aab5-f8d2b2bc95a1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:07:59.496968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.496895    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7kb7\" (UniqueName: \"kubernetes.io/projected/5802a4b8-1826-4452-aab5-f8d2b2bc95a1-kube-api-access-h7kb7\") pod \"llmisvc-controller-manager-6466c554d8-5xbhb\" (UID: \"5802a4b8-1826-4452-aab5-f8d2b2bc95a1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:07:59.597682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.597649    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5802a4b8-1826-4452-aab5-f8d2b2bc95a1-cert\") pod \"llmisvc-controller-manager-6466c554d8-5xbhb\" (UID: \"5802a4b8-1826-4452-aab5-f8d2b2bc95a1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:07:59.598073 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.597769    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7kb7\" (UniqueName: \"kubernetes.io/projected/5802a4b8-1826-4452-aab5-f8d2b2bc95a1-kube-api-access-h7kb7\") pod \"llmisvc-controller-manager-6466c554d8-5xbhb\" (UID: \"5802a4b8-1826-4452-aab5-f8d2b2bc95a1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:07:59.600382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.600356    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5802a4b8-1826-4452-aab5-f8d2b2bc95a1-cert\") pod \"llmisvc-controller-manager-6466c554d8-5xbhb\" (UID: \"5802a4b8-1826-4452-aab5-f8d2b2bc95a1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:07:59.611237 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.611213    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7kb7\" (UniqueName: \"kubernetes.io/projected/5802a4b8-1826-4452-aab5-f8d2b2bc95a1-kube-api-access-h7kb7\") pod \"llmisvc-controller-manager-6466c554d8-5xbhb\" (UID: \"5802a4b8-1826-4452-aab5-f8d2b2bc95a1\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:07:59.646247 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.646182    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4"
Oct 31 13:07:59.646247 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.646199    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lkmx4" event={"ID":"caac3829-d20c-45c1-ac3f-33b36f5afd6a","Type":"ContainerDied","Data":"a31dab26a9b72683dcfcdfb1985a34612409c9fb687e6c46694c0fa301a06cb5"}
Oct 31 13:07:59.646247 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.646241    5787 scope.go:122] "RemoveContainer" containerID="f0041c915d6fdc7cdb2aabe1c357a34ea4bcd08343aa868c87714e2c282cc94f"
Oct 31 13:07:59.669865 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.669839    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lkmx4"]
Oct 31 13:07:59.673748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.673728    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lkmx4"]
Oct 31 13:07:59.699976 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.699953    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:07:59.974763 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:07:59.974733    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"]
Oct 31 13:07:59.977359 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:07:59.977334    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5802a4b8_1826_4452_aab5_f8d2b2bc95a1.slice/crio-f777fab3619e642d59bdc129cf9577a7d1ceee47f7414d3f3ea92db526025fa5 WatchSource:0}: Error finding container f777fab3619e642d59bdc129cf9577a7d1ceee47f7414d3f3ea92db526025fa5: Status 404 returned error can't find the container with id f777fab3619e642d59bdc129cf9577a7d1ceee47f7414d3f3ea92db526025fa5
Oct 31 13:08:00.653302 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:00.653268    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb" event={"ID":"5802a4b8-1826-4452-aab5-f8d2b2bc95a1","Type":"ContainerStarted","Data":"33df49dbe164a2d83b95ad88e3c2f536e4f9e2172c8f3f3740276f064785c158"}
Oct 31 13:08:00.653721 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:00.653324    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb" event={"ID":"5802a4b8-1826-4452-aab5-f8d2b2bc95a1","Type":"ContainerStarted","Data":"f777fab3619e642d59bdc129cf9577a7d1ceee47f7414d3f3ea92db526025fa5"}
Oct 31 13:08:00.653721 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:00.653426    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:08:00.703829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:00.703746    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caac3829-d20c-45c1-ac3f-33b36f5afd6a" path="/var/lib/kubelet/pods/caac3829-d20c-45c1-ac3f-33b36f5afd6a/volumes"
Oct 31 13:08:18.245726 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:08:18.245665    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:08:31.659919 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:31.659891    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb"
Oct 31 13:08:31.683162 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:31.683105    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-5xbhb" podStartSLOduration=32.22394001 podStartE2EDuration="32.683090577s" totalImagesPullingTime="459.150567ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:07:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:08:00.675181823 +0000 UTC m=+460.091123663" watchObservedRunningTime="2026-10-31 13:08:31.683090577 +0000 UTC m=+491.099032416"
Oct 31 13:08:31.713423 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:31.713387    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"]
Oct 31 13:08:31.713617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:31.713593    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz" podUID="f6eef8c9-3f0a-4560-b20d-7bf495c771b8" containerName="manager" containerID="cri-o://884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c" gracePeriod=30
Oct 31 13:08:31.955167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:31.955142    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:08:32.071150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.071121    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-kube-api-access-kpx9b\" (UniqueName: \"kubernetes.io/projected/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-kube-api-access-kpx9b\") pod \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\" (UID: \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\") "
Oct 31 13:08:32.071333 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.071191    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-cert\" (UniqueName: \"kubernetes.io/secret/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-cert\") pod \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\" (UID: \"f6eef8c9-3f0a-4560-b20d-7bf495c771b8\") "
Oct 31 13:08:32.073492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.073461    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-kube-api-access-kpx9b" pod "f6eef8c9-3f0a-4560-b20d-7bf495c771b8" (UID: "f6eef8c9-3f0a-4560-b20d-7bf495c771b8"). InnerVolumeSpecName "kube-api-access-kpx9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:08:32.073602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.073464    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-cert" pod "f6eef8c9-3f0a-4560-b20d-7bf495c771b8" (UID: "f6eef8c9-3f0a-4560-b20d-7bf495c771b8"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:08:32.172666 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.172626    5787 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:08:32.172666 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.172662    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpx9b\" (UniqueName: \"kubernetes.io/projected/f6eef8c9-3f0a-4560-b20d-7bf495c771b8-kube-api-access-kpx9b\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:08:32.798901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.798870    5787 generic.go:356] "Generic (PLEG): container finished" podID="f6eef8c9-3f0a-4560-b20d-7bf495c771b8" containerID="884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c" exitCode=0
Oct 31 13:08:32.799325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.798919    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"
Oct 31 13:08:32.799325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.798959    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz" event={"ID":"f6eef8c9-3f0a-4560-b20d-7bf495c771b8","Type":"ContainerDied","Data":"884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c"}
Oct 31 13:08:32.799325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.799012    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz" event={"ID":"f6eef8c9-3f0a-4560-b20d-7bf495c771b8","Type":"ContainerDied","Data":"f49bdc60f36842aec82f38108e251875f0850733e674fed66330c1b459b33acd"}
Oct 31 13:08:32.799325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.799036    5787 scope.go:122] "RemoveContainer" containerID="884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c"
Oct 31 13:08:32.807394 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.807374    5787 scope.go:122] "RemoveContainer" containerID="884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c"
Oct 31 13:08:32.807680 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:08:32.807664    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c\": container with ID starting with 884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c not found: ID does not exist" containerID="884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c"
Oct 31 13:08:32.807777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.807690    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c"} err="failed to get container status \"884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c\": rpc error: code = NotFound desc = could not find container \"884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c\": container with ID starting with 884dd141f34720db6f0c89a818a8cf177f515bbba19263ad48089ddadc10c79c not found: ID does not exist"
Oct 31 13:08:32.820902 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.820868    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"]
Oct 31 13:08:32.840491 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:32.840465    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-l2rhz"]
Oct 31 13:08:34.703523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:08:34.703494    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6eef8c9-3f0a-4560-b20d-7bf495c771b8" path="/var/lib/kubelet/pods/f6eef8c9-3f0a-4560-b20d-7bf495c771b8/volumes"
Oct 31 13:08:48.245980 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:08:48.245937    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:09:18.245881 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:09:18.245781    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:09:25.217537 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.217495    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"]
Oct 31 13:09:25.218348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.218314    5787 cpu_manager.go:414] "containerMap: removing container" podUID="f6eef8c9-3f0a-4560-b20d-7bf495c771b8" containerName="manager"
Oct 31 13:09:25.218348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.218345    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6eef8c9-3f0a-4560-b20d-7bf495c771b8" containerName="manager"
Oct 31 13:09:25.218538 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.218449    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6eef8c9-3f0a-4560-b20d-7bf495c771b8" containerName="manager"
Oct 31 13:09:25.221127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.221103    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.266385 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.266353    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"]
Oct 31 13:09:25.290361 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.290335    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dmjbc\""
Oct 31 13:09:25.345553 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.345521    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95c8381-0aff-448c-9422-d3693062872f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.345754 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.345567    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.345754 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.345625    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95c8381-0aff-448c-9422-d3693062872f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.345754 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.345679    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjss5\" (UniqueName: \"kubernetes.io/projected/d95c8381-0aff-448c-9422-d3693062872f-kube-api-access-hjss5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.345754 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.345734    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.345973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.345808    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.345973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.345858    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.345973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.345923    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.345973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.345961    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95c8381-0aff-448c-9422-d3693062872f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447034    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447074    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447100    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447130    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95c8381-0aff-448c-9422-d3693062872f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447163    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95c8381-0aff-448c-9422-d3693062872f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447197    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447223    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95c8381-0aff-448c-9422-d3693062872f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447263    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjss5\" (UniqueName: \"kubernetes.io/projected/d95c8381-0aff-448c-9422-d3693062872f-kube-api-access-hjss5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447289    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447551    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447665    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447736    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.447857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447850    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.448119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.447934    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d95c8381-0aff-448c-9422-d3693062872f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.449588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.449566    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d95c8381-0aff-448c-9422-d3693062872f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.450016 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.449999    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d95c8381-0aff-448c-9422-d3693062872f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.461660 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.461630    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d95c8381-0aff-448c-9422-d3693062872f-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.462897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.462747    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjss5\" (UniqueName: \"kubernetes.io/projected/d95c8381-0aff-448c-9422-d3693062872f-kube-api-access-hjss5\") pod \"maas-default-gateway-openshift-default-7f5667f86c-rz8cm\" (UID: \"d95c8381-0aff-448c-9422-d3693062872f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.533051 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.532957    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:25.825160 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:25.825136    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"]
Oct 31 13:09:25.826784 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:09:25.826760    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95c8381_0aff_448c_9422_d3693062872f.slice/crio-8c3f4574ced6c66252285dfc92c58ebbdc6429270ea4d5f770d6ebfb6e6d7d22 WatchSource:0}: Error finding container 8c3f4574ced6c66252285dfc92c58ebbdc6429270ea4d5f770d6ebfb6e6d7d22: Status 404 returned error can't find the container with id 8c3f4574ced6c66252285dfc92c58ebbdc6429270ea4d5f770d6ebfb6e6d7d22
Oct 31 13:09:26.029535 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:26.029502    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" event={"ID":"d95c8381-0aff-448c-9422-d3693062872f","Type":"ContainerStarted","Data":"8c3f4574ced6c66252285dfc92c58ebbdc6429270ea4d5f770d6ebfb6e6d7d22"}
Oct 31 13:09:28.559035 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:28.559004    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:09:28.559324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:28.559136    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:09:28.559324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:28.559172    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:09:29.045800 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:29.045764    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" event={"ID":"d95c8381-0aff-448c-9422-d3693062872f","Type":"ContainerStarted","Data":"88e55ccded19706ba7feb890b4f342b435c8c565d2b3cad7a5e0527544ffce1c"}
Oct 31 13:09:29.077781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:29.077730    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podStartSLOduration=1.348254647 podStartE2EDuration="4.077697494s" totalImagesPullingTime="2.729442847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:29.075235577 +0000 UTC m=+548.491177416" watchObservedRunningTime="2026-10-31 13:09:29.077697494 +0000 UTC m=+548.493639328"
Oct 31 13:09:29.552099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:29.552065    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:29.559898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:29.559852    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:09:29.560400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:29.559939    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 31 13:09:30.539291 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:30.539086    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:09:30.539577 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:30.539546    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 31 13:09:31.533370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:31.533329    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:09:31.533924 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:31.533402    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 31 13:09:32.554732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:32.547165    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:09:32.554732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:32.547245    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 31 13:09:33.543731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:33.534068    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:09:33.543731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:33.534129    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 31 13:09:34.533499 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:34.533466    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:09:34.534007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:34.533540    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 31 13:09:35.534075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:35.534035    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Oct 31 13:09:35.534466 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:35.534120    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Oct 31 13:09:37.534815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:37.534774    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:09:37.535254 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:37.534846    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:09:38.577300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:38.577047    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:09:38.578099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:38.578020    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:09:38.945191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:38.945160    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:38.945503 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:38.945469    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:38.946465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:09:38.946443    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm"
Oct 31 13:09:48.245721 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:09:48.245660    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:10:18.245092 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:10:18.245053    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:10:21.607942 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.607904    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"]
Oct 31 13:10:21.611553 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.611530    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.628626 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.628582    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"]
Oct 31 13:10:21.636209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.636173    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-79c6ff-openshift-default-dockercfg-tkwch\""
Oct 31 13:10:21.707960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.707929    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"]
Oct 31 13:10:21.711470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.711447    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.726829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.726794    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"]
Oct 31 13:10:21.736375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.736346    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-004f64-openshift-default-dockercfg-k4xnk\""
Oct 31 13:10:21.765724 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.765675    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.765921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.765793    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.765921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.765875    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.766029 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.765936    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.766130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766046    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.766130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766090    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.766288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766265    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.766422 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766405    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca15df2-3a97-4142-8a38-8a0053421cf7-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.766470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766447    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d9a104-017d-48ce-8831-5d9e0635b637-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.766507 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766491    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42xv9\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-kube-api-access-42xv9\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.766541 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766512    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.766584 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766567    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.766618 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766593    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.766618 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766608    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d9a104-017d-48ce-8831-5d9e0635b637-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.766682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766672    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.766739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766691    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.766781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766770    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjbvw\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-kube-api-access-vjbvw\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.766820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.766799    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.867921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.867825    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjbvw\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-kube-api-access-vjbvw\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.867921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.867865    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.867921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.867900    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.868207 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868093    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.868207 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868128    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.868320 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868295    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.868377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868354    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.868434 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868388    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.868434 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868414    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.868539 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868457    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.868539 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868469    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.868647 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868546    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca15df2-3a97-4142-8a38-8a0053421cf7-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.868647 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868591    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d9a104-017d-48ce-8831-5d9e0635b637-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.868779 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868649    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868955    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42xv9\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-kube-api-access-42xv9\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.868999    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869044    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869063    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869068    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869302    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d9a104-017d-48ce-8831-5d9e0635b637-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869321    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869350    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869375    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869411    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca15df2-3a97-4142-8a38-8a0053421cf7-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.869799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869506    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.870436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.869914    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.870436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.870090    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.870436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.870120    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d9a104-017d-48ce-8831-5d9e0635b637-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.871092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.871070    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.871558 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.871521    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d9a104-017d-48ce-8831-5d9e0635b637-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.872337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.872315    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.872913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.872888    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.882352 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.882306    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.882506 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.882400    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42xv9\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-kube-api-access-42xv9\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:21.882506 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.882416    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjbvw\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-kube-api-access-vjbvw\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.882773 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.882752    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:21.925926 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:21.925888    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:22.029115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.028667    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:22.250528 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.250487    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"]
Oct 31 13:10:22.254920 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:10:22.254872    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcca15df2_3a97_4142_8a38_8a0053421cf7.slice/crio-8585f7ed7f2646daee016f2c2280ea756543e2c899217be60049e8124444bc48 WatchSource:0}: Error finding container 8585f7ed7f2646daee016f2c2280ea756543e2c899217be60049e8124444bc48: Status 404 returned error can't find the container with id 8585f7ed7f2646daee016f2c2280ea756543e2c899217be60049e8124444bc48
Oct 31 13:10:22.257717 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.257667    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:22.257823 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.257753    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:22.257823 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.257783    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:22.296980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.296944    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" event={"ID":"cca15df2-3a97-4142-8a38-8a0053421cf7","Type":"ContainerStarted","Data":"8585f7ed7f2646daee016f2c2280ea756543e2c899217be60049e8124444bc48"}
Oct 31 13:10:22.323551 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.323298    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"]
Oct 31 13:10:22.327243 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:10:22.327196    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d9a104_017d_48ce_8831_5d9e0635b637.slice/crio-7b91d7962b9ec9aadfb81f78e640002e6d8e1d3d796deb220abf394645e37836 WatchSource:0}: Error finding container 7b91d7962b9ec9aadfb81f78e640002e6d8e1d3d796deb220abf394645e37836: Status 404 returned error can't find the container with id 7b91d7962b9ec9aadfb81f78e640002e6d8e1d3d796deb220abf394645e37836
Oct 31 13:10:22.330298 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.330264    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:22.330398 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.330336    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:22.330398 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:22.330364    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:23.304174 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:23.304132    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" event={"ID":"07d9a104-017d-48ce-8831-5d9e0635b637","Type":"ContainerStarted","Data":"82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5"}
Oct 31 13:10:23.304652 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:23.304200    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" event={"ID":"07d9a104-017d-48ce-8831-5d9e0635b637","Type":"ContainerStarted","Data":"7b91d7962b9ec9aadfb81f78e640002e6d8e1d3d796deb220abf394645e37836"}
Oct 31 13:10:23.305816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:23.305789    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" event={"ID":"cca15df2-3a97-4142-8a38-8a0053421cf7","Type":"ContainerStarted","Data":"b0023782a8c0b6b535aa17d10da902c9548ba2bc786c6a4cb138074a85a641d8"}
Oct 31 13:10:23.330954 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:23.330898    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" podStartSLOduration=2.33088359 podStartE2EDuration="2.33088359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:23.329495894 +0000 UTC m=+602.745437736" watchObservedRunningTime="2026-10-31 13:10:23.33088359 +0000 UTC m=+602.746825457"
Oct 31 13:10:23.362232 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:23.362168    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podStartSLOduration=2.362152409 podStartE2EDuration="2.362152409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:23.360627549 +0000 UTC m=+602.776569384" watchObservedRunningTime="2026-10-31 13:10:23.362152409 +0000 UTC m=+602.778094245"
Oct 31 13:10:23.927091 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:23.927053    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:23.933303 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:23.933277    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:24.029722 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:24.029661    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:24.035797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:24.035767    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:24.311310 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:24.311264    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:24.311773 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:24.311324    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:24.312158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:24.312140    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:24.312205 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:24.312193    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:30.473440 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.473403    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"]
Oct 31 13:10:30.477032 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.477011    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:30.485939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.485841    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-004f64\""
Oct 31 13:10:30.486441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.486422    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-004f64-dockercfg-p54kt\""
Oct 31 13:10:30.487462 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.487437    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"]
Oct 31 13:10:30.530084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.530046    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"]
Oct 31 13:10:30.533528 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.533510    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:30.546211 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.546179    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"]
Oct 31 13:10:30.576334 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.576304    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786\" (UID: \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:30.576334 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.576337    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqq7h\" (UniqueName: \"kubernetes.io/projected/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-kube-api-access-qqq7h\") pod \"payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786\" (UID: \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:30.677221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.677186    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x\" (UID: \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:30.677385 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.677242    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786\" (UID: \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:30.677385 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.677353    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqq7h\" (UniqueName: \"kubernetes.io/projected/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-kube-api-access-qqq7h\") pod \"payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786\" (UID: \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:30.677495 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.677396    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrc5s\" (UniqueName: \"kubernetes.io/projected/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-kube-api-access-zrc5s\") pod \"payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x\" (UID: \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:30.677953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.677935    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786\" (UID: \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:30.693489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.693455    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqq7h\" (UniqueName: \"kubernetes.io/projected/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-kube-api-access-qqq7h\") pod \"payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786\" (UID: \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:30.777984 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.777884    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrc5s\" (UniqueName: \"kubernetes.io/projected/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-kube-api-access-zrc5s\") pod \"payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x\" (UID: \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:30.778155 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.777985    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x\" (UID: \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:30.778694 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.778674    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x\" (UID: \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:30.792528 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.792499    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:30.793090 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.793068    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrc5s\" (UniqueName: \"kubernetes.io/projected/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-kube-api-access-zrc5s\") pod \"payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x\" (UID: \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:30.849789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:30.849760    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:31.072428 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:31.072399    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"]
Oct 31 13:10:31.074725 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:10:31.074680    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cda9f0_fd0c_4fc0_9c27_3435ea9dec75.slice/crio-f7f58a190fa89b3ee6f275a5f46b46e2b27d25db1296724b074237dc7dd3bbbd WatchSource:0}: Error finding container f7f58a190fa89b3ee6f275a5f46b46e2b27d25db1296724b074237dc7dd3bbbd: Status 404 returned error can't find the container with id f7f58a190fa89b3ee6f275a5f46b46e2b27d25db1296724b074237dc7dd3bbbd
Oct 31 13:10:31.140328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:31.140303    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"]
Oct 31 13:10:31.142638 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:10:31.142608    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6893107_8ce0_40a6_846e_a7eecfdbe6f5.slice/crio-0f1d91241dd7ab65dec5fed68964337d96dbbe5ecfe757c523e758c4ec4554f5 WatchSource:0}: Error finding container 0f1d91241dd7ab65dec5fed68964337d96dbbe5ecfe757c523e758c4ec4554f5: Status 404 returned error can't find the container with id 0f1d91241dd7ab65dec5fed68964337d96dbbe5ecfe757c523e758c4ec4554f5
Oct 31 13:10:31.351855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:31.351748    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786" event={"ID":"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75","Type":"ContainerStarted","Data":"f7f58a190fa89b3ee6f275a5f46b46e2b27d25db1296724b074237dc7dd3bbbd"}
Oct 31 13:10:31.353137 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:31.353106    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x" event={"ID":"a6893107-8ce0-40a6-846e-a7eecfdbe6f5","Type":"ContainerStarted","Data":"0f1d91241dd7ab65dec5fed68964337d96dbbe5ecfe757c523e758c4ec4554f5"}
Oct 31 13:10:34.372894 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:34.372848    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x" event={"ID":"a6893107-8ce0-40a6-846e-a7eecfdbe6f5","Type":"ContainerStarted","Data":"e6eb4fb8f1b406c7f5f605af2bf7474635fb42f63b1f2316f4cfae7a4e6b7d90"}
Oct 31 13:10:34.373399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:34.372935    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:34.375145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:34.375117    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786" event={"ID":"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75","Type":"ContainerStarted","Data":"7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236"}
Oct 31 13:10:34.375295 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:34.375201    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:34.393474 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:34.393434    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x" podStartSLOduration=1.898490055 podStartE2EDuration="4.393422196s" totalImagesPullingTime="2.494932141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:34.393132806 +0000 UTC m=+613.809074645" watchObservedRunningTime="2026-10-31 13:10:34.393422196 +0000 UTC m=+613.809364030"
Oct 31 13:10:34.413373 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:34.413322    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786" podStartSLOduration=1.820214806 podStartE2EDuration="4.413307455s" totalImagesPullingTime="2.593092649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:34.411195358 +0000 UTC m=+613.827137199" watchObservedRunningTime="2026-10-31 13:10:34.413307455 +0000 UTC m=+613.829249294"
Oct 31 13:10:42.389956 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.389911    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"]
Oct 31 13:10:42.397157 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.397125    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.404119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.404060    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"]
Oct 31 13:10:42.404362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.404135    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"openshift-service-ca.crt\""
Oct 31 13:10:42.404887 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.404866    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"default-dockercfg-s79md\""
Oct 31 13:10:42.406307 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.406283    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"simulator-w4-426787-kserve-self-signed-certs\""
Oct 31 13:10:42.411984 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.411964    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blvdm\" (UniqueName: \"kubernetes.io/projected/ffd95bfe-b694-41a9-bf39-ad654cedee39-kube-api-access-blvdm\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.412082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.411999    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-home\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.412082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.412042    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-dshm\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.412082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.412078    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-model-cache\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.412186 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.412099    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd95bfe-b694-41a9-bf39-ad654cedee39-tls-certs\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.412186 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.412120    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-tmp-dir\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.506745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.503807    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"kube-root-ca.crt\""
Oct 31 13:10:42.514192 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.514156    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd95bfe-b694-41a9-bf39-ad654cedee39-tls-certs\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.514382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.514204    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-tmp-dir\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.514382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.514231    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blvdm\" (UniqueName: \"kubernetes.io/projected/ffd95bfe-b694-41a9-bf39-ad654cedee39-kube-api-access-blvdm\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.514382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.514259    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-home\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.514382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.514307    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-dshm\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.514382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.514347    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-model-cache\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.514695 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.514674    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-model-cache\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.515849 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.515827    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-tmp-dir\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.517045 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.517022    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-home\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.519382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.519362    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd95bfe-b694-41a9-bf39-ad654cedee39-tls-certs\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.521367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.521315    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-dshm\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.530848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.530817    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blvdm\" (UniqueName: \"kubernetes.io/projected/ffd95bfe-b694-41a9-bf39-ad654cedee39-kube-api-access-blvdm\") pod \"simulator-w4-426787-kserve-588f6d7647-wkvg6\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") " pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:42.714416 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:42.714374    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:43.056788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.050097    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"]
Oct 31 13:10:43.067015 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:10:43.062120    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffd95bfe_b694_41a9_bf39_ad654cedee39.slice/crio-e9896c90e2442d40ec7ccc091a64e8d1ed67cf2e62e1b1a8bbd7b090d7b953cb WatchSource:0}: Error finding container e9896c90e2442d40ec7ccc091a64e8d1ed67cf2e62e1b1a8bbd7b090d7b953cb: Status 404 returned error can't find the container with id e9896c90e2442d40ec7ccc091a64e8d1ed67cf2e62e1b1a8bbd7b090d7b953cb
Oct 31 13:10:43.432339 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.432251    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" event={"ID":"ffd95bfe-b694-41a9-bf39-ad654cedee39","Type":"ContainerStarted","Data":"e9896c90e2442d40ec7ccc091a64e8d1ed67cf2e62e1b1a8bbd7b090d7b953cb"}
Oct 31 13:10:43.776768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.776079    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"]
Oct 31 13:10:43.803221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.802415    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"]
Oct 31 13:10:43.803221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.802550    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.814992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.814718    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f9e24f\"/\"default-dockercfg-fs68n\""
Oct 31 13:10:43.814992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.814818    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f9e24f\"/\"simulator-w5-f9e24f-kserve-self-signed-certs\""
Oct 31 13:10:43.814992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.814885    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f9e24f\"/\"kube-root-ca.crt\""
Oct 31 13:10:43.814992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.814944    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-f9e24f\"/\"openshift-service-ca.crt\""
Oct 31 13:10:43.831592 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.831529    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-dshm\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.831782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.831602    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8frn\" (UniqueName: \"kubernetes.io/projected/c43584fa-29e8-4ab5-ab4e-366ed24bade8-kube-api-access-c8frn\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.831782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.831660    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tmp-dir\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.831920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.831819    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-home\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.831920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.831859    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-model-cache\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.832022 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.831934    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tls-certs\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.933739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.933015    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tls-certs\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.933739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.933085    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-dshm\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.933739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.933144    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8frn\" (UniqueName: \"kubernetes.io/projected/c43584fa-29e8-4ab5-ab4e-366ed24bade8-kube-api-access-c8frn\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.933739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.933197    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tmp-dir\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.933739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.933238    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-home\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.933739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.933270    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-model-cache\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.935719 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.934792    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-model-cache\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.935719 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.935639    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-home\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.936361 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.936068    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tmp-dir\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.961750 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.958022    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-dshm\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.961750 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.959299    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tls-certs\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:43.966571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:43.966474    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8frn\" (UniqueName: \"kubernetes.io/projected/c43584fa-29e8-4ab5-ab4e-366ed24bade8-kube-api-access-c8frn\") pod \"simulator-w5-f9e24f-kserve-684b9b49bd-rp99v\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") " pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:44.119989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:44.119886    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:44.572733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:44.561328    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"]
Oct 31 13:10:45.399734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:45.399448    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:10:45.401618 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:45.400976    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:10:45.513621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:45.513555    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" event={"ID":"c43584fa-29e8-4ab5-ab4e-366ed24bade8","Type":"ContainerStarted","Data":"b3cdf3d13698c5174fc1396a3f16f608ed618f0cd2bb558919eede941a3ed53c"}
Oct 31 13:10:46.708448 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.708330    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"]
Oct 31 13:10:46.712083 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.711992    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.721053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.720991    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"]
Oct 31 13:10:46.725983 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.725954    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"default-dockercfg-nn5dd\""
Oct 31 13:10:46.726579 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.726557    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"simulator-w0-7bf66c-kserve-self-signed-certs\""
Oct 31 13:10:46.726768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.726752    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"openshift-service-ca.crt\""
Oct 31 13:10:46.727227 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.727146    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7bf66c\"/\"kube-root-ca.crt\""
Oct 31 13:10:46.829156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.829119    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-model-cache\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.829366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.829177    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-home\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.829366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.829218    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcswr\" (UniqueName: \"kubernetes.io/projected/debe71f3-91f0-48ac-b79b-85472ff7defd-kube-api-access-rcswr\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.829366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.829300    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/debe71f3-91f0-48ac-b79b-85472ff7defd-tls-certs\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.829366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.829332    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-dshm\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.829589 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.829375    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-tmp-dir\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.934424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.932567    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-home\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.934424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.932626    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcswr\" (UniqueName: \"kubernetes.io/projected/debe71f3-91f0-48ac-b79b-85472ff7defd-kube-api-access-rcswr\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.934424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.932735    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/debe71f3-91f0-48ac-b79b-85472ff7defd-tls-certs\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.934424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.932775    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-dshm\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.934424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.932820    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-tmp-dir\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.934424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.932917    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-model-cache\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.934424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.933359    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-model-cache\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.934424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.933637    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-home\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.935461 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.935437    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-tmp-dir\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.958378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.944644    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/debe71f3-91f0-48ac-b79b-85472ff7defd-tls-certs\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.958378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.947249    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-dshm\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:46.958378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:46.948660    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcswr\" (UniqueName: \"kubernetes.io/projected/debe71f3-91f0-48ac-b79b-85472ff7defd-kube-api-access-rcswr\") pod \"simulator-w0-7bf66c-kserve-55564fdfb9-m28nd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") " pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:47.032445 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:47.032216    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:47.353264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:47.353178    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"]
Oct 31 13:10:47.357850 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:10:47.357810    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddebe71f3_91f0_48ac_b79b_85472ff7defd.slice/crio-206b239f623536772d67803aab123a1eb162734b06a60f17c99596522b76c27d WatchSource:0}: Error finding container 206b239f623536772d67803aab123a1eb162734b06a60f17c99596522b76c27d: Status 404 returned error can't find the container with id 206b239f623536772d67803aab123a1eb162734b06a60f17c99596522b76c27d
Oct 31 13:10:47.536670 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:47.536636    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" event={"ID":"ffd95bfe-b694-41a9-bf39-ad654cedee39","Type":"ContainerStarted","Data":"0ad6d5ed06c872565a89df2be308d475f2176f71d27455fdcf1738851aa1b748"}
Oct 31 13:10:47.539024 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:47.538961    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" event={"ID":"debe71f3-91f0-48ac-b79b-85472ff7defd","Type":"ContainerStarted","Data":"3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55"}
Oct 31 13:10:47.539024 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:47.538998    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" event={"ID":"debe71f3-91f0-48ac-b79b-85472ff7defd","Type":"ContainerStarted","Data":"206b239f623536772d67803aab123a1eb162734b06a60f17c99596522b76c27d"}
Oct 31 13:10:47.562088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:47.562036    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podStartSLOduration=1.665344537 podStartE2EDuration="5.562021418s" totalImagesPullingTime="3.896676881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:47.559364006 +0000 UTC m=+626.975305843" watchObservedRunningTime="2026-10-31 13:10:47.562021418 +0000 UTC m=+626.977963257"
Oct 31 13:10:47.583668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:47.583501    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podStartSLOduration=1.5834799099999999 podStartE2EDuration="1.58347991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:47.581871578 +0000 UTC m=+626.997813419" watchObservedRunningTime="2026-10-31 13:10:47.58347991 +0000 UTC m=+626.999421746"
Oct 31 13:10:47.715401 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:47.715364    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:48.033266 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:48.033180    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:48.247946 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:10:48.247906    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:10:48.546495 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:48.546423    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" event={"ID":"c43584fa-29e8-4ab5-ab4e-366ed24bade8","Type":"ContainerStarted","Data":"2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0"}
Oct 31 13:10:48.571277 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:48.571201    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podStartSLOduration=2.691743605 podStartE2EDuration="5.571178763s" totalImagesPullingTime="2.879435158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:48.570485346 +0000 UTC m=+627.986427190" watchObservedRunningTime="2026-10-31 13:10:48.571178763 +0000 UTC m=+627.987120604"
Oct 31 13:10:49.120538 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:49.120501    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:50.560715 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.560617    5787 generic.go:356] "Generic (PLEG): container finished" podID="07d9a104-017d-48ce-8831-5d9e0635b637" containerID="82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5" exitCode=0
Oct 31 13:10:50.561225 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.560687    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" event={"ID":"07d9a104-017d-48ce-8831-5d9e0635b637","Type":"ContainerDied","Data":"82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5"}
Oct 31 13:10:50.561225 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.561131    5787 scope.go:122] "RemoveContainer" containerID="82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5"
Oct 31 13:10:50.562782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.562746    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:50.562874 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.562846    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:50.562930 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.562901    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:50.563789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.563769    5787 generic.go:356] "Generic (PLEG): container finished" podID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerID="b0023782a8c0b6b535aa17d10da902c9548ba2bc786c6a4cb138074a85a641d8" exitCode=0
Oct 31 13:10:50.563912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.563804    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" event={"ID":"cca15df2-3a97-4142-8a38-8a0053421cf7","Type":"ContainerDied","Data":"b0023782a8c0b6b535aa17d10da902c9548ba2bc786c6a4cb138074a85a641d8"}
Oct 31 13:10:50.564288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.564263    5787 scope.go:122] "RemoveContainer" containerID="b0023782a8c0b6b535aa17d10da902c9548ba2bc786c6a4cb138074a85a641d8"
Oct 31 13:10:50.565630 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.565598    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:50.565734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.565683    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:50.565824 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:50.565763    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:10:51.577580 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:51.577531    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" event={"ID":"07d9a104-017d-48ce-8831-5d9e0635b637","Type":"ContainerStarted","Data":"6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e"}
Oct 31 13:10:51.581746 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:51.581026    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" event={"ID":"cca15df2-3a97-4142-8a38-8a0053421cf7","Type":"ContainerStarted","Data":"09bc8226dd4192f71df94f9456b128de5c8713362abc2a269e8eb53744eebd18"}
Oct 31 13:10:51.926880 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:51.926795    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:51.928179 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:51.928149    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 13:10:51.928360 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:51.928209    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 13:10:52.029768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:52.029729    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:52.031047 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:52.031005    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 13:10:52.031191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:52.031081    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 13:10:52.715189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:52.715164    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:52.728675 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:52.728648    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:52.926375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:52.926335    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 13:10:52.926563 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:52.926394    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 13:10:53.029304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:53.029204    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 13:10:53.029304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:53.029270    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 13:10:53.611880 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:53.611852    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:10:53.926960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:53.926865    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 13:10:53.926960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:53.926928    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 13:10:54.029287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:54.029253    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 13:10:54.029468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:54.029305    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 13:10:54.120888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:54.120852    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:54.139076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:54.139042    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:54.312062 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:54.312025    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:54.312771 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:54.312749    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:54.628340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:54.628262    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:10:54.927092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:54.927010    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 13:10:54.927092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:54.927066    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 13:10:55.029751 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:55.029685    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 13:10:55.029927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:55.029763    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 13:10:55.927263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:55.927225    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 13:10:55.927670 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:55.927284    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 13:10:56.029871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.029835    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 13:10:56.030065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.029891    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 13:10:56.898275 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.898237    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"]
Oct 31 13:10:56.905333 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.905292    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:56.912193 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.911824    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"openshift-service-ca.crt\""
Oct 31 13:10:56.912419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.911837    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"kube-root-ca.crt\""
Oct 31 13:10:56.912851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.912827    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"]
Oct 31 13:10:56.913032 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.912991    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"default-dockercfg-szbg5\""
Oct 31 13:10:56.913177 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.913159    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-2a262a\"/\"simulator-w2-2a262a-kserve-self-signed-certs\""
Oct 31 13:10:56.926439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.926408    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Oct 31 13:10:56.926599 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.926465    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Oct 31 13:10:56.999446 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:56.999398    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqqvd\" (UniqueName: \"kubernetes.io/projected/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-kube-api-access-bqqvd\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.000112 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.000084    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-dshm\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.000241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.000226    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-home\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.000354 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.000339    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tls-certs\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.000486 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.000472    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-model-cache\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.000636 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.000620    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tmp-dir\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.029974 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.029939    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 31 13:10:57.030215 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.030192    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 31 13:10:57.032776 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.032751    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:57.086563 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.086531    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:57.101403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.101362    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqqvd\" (UniqueName: \"kubernetes.io/projected/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-kube-api-access-bqqvd\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.101612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.101516    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-dshm\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.101612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.101557    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-home\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.101612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.101590    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tls-certs\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.101817 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.101636    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-model-cache\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.101817 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.101725    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tmp-dir\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.102241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.102213    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-home\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.102425 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.102245    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tmp-dir\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.102537 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.102487    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-model-cache\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.104989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.104967    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tls-certs\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.105584 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.105545    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-dshm\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.117988 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.117958    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqqvd\" (UniqueName: \"kubernetes.io/projected/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-kube-api-access-bqqvd\") pod \"simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") " pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.225964 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.225925    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:10:57.576327 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.576301    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"]
Oct 31 13:10:57.579148 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:10:57.579115    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7416b2b0_1a8a_4887_aec9_cc7592a5de9e.slice/crio-ad22bc0f83881b62e636dfddba92a4b41bfbb7bd864f83a892a898eb5c68ebc7 WatchSource:0}: Error finding container ad22bc0f83881b62e636dfddba92a4b41bfbb7bd864f83a892a898eb5c68ebc7: Status 404 returned error can't find the container with id ad22bc0f83881b62e636dfddba92a4b41bfbb7bd864f83a892a898eb5c68ebc7
Oct 31 13:10:57.635957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.635914    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" event={"ID":"7416b2b0-1a8a-4887-aec9-cc7592a5de9e","Type":"ContainerStarted","Data":"ad22bc0f83881b62e636dfddba92a4b41bfbb7bd864f83a892a898eb5c68ebc7"}
Oct 31 13:10:57.651418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.649991    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:10:57.933730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.933609    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:57.934902 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:57.934869    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:10:58.036155 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:58.036124    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:58.038673 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:58.038651    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:10:58.650006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:58.649972    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" event={"ID":"7416b2b0-1a8a-4887-aec9-cc7592a5de9e","Type":"ContainerStarted","Data":"8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d"}
Oct 31 13:10:58.672694 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:58.672623    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podStartSLOduration=2.6726039200000002 podStartE2EDuration="2.67260392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:58.671452104 +0000 UTC m=+638.087393944" watchObservedRunningTime="2026-10-31 13:10:58.67260392 +0000 UTC m=+638.088545760"
Oct 31 13:10:59.227906 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:10:59.227861    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:11:05.373838 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.373747    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"]
Oct 31 13:11:05.386465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.386433    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.394143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.394106    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"]
Oct 31 13:11:05.395310 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.395283    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-004f64\"/\"simulator-w3-004f64-kserve-self-signed-certs\""
Oct 31 13:11:05.395831 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.395513    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-004f64\"/\"kube-root-ca.crt\""
Oct 31 13:11:05.395915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.395828    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-004f64\"/\"default-dockercfg-qqb6j\""
Oct 31 13:11:05.395915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.395735    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-004f64\"/\"openshift-service-ca.crt\""
Oct 31 13:11:05.403545 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.403513    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8ls5\" (UniqueName: \"kubernetes.io/projected/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-kube-api-access-d8ls5\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.403731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.403567    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-model-cache\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.403731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.403719    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tmp-dir\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.403871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.403795    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-home\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.403938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.403921    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tls-certs\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.404036 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.404018    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-dshm\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.504925 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.504880    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tls-certs\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.505143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.504945    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-dshm\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.505143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.504981    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8ls5\" (UniqueName: \"kubernetes.io/projected/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-kube-api-access-d8ls5\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.505143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.505009    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-model-cache\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.505143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.505044    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tmp-dir\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.505143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.505084    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-home\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.505553 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.505509    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-home\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.505553 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.505538    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tmp-dir\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.505780 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.505602    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-model-cache\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.507581 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.507554    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-dshm\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.507911 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.507889    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tls-certs\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.519107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.519079    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8ls5\" (UniqueName: \"kubernetes.io/projected/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-kube-api-access-d8ls5\") pod \"simulator-w3-004f64-kserve-54dd474994-4xzlb\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") " pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:05.704748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:05.704614    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:06.000651 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:06.000628    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"]
Oct 31 13:11:06.003583 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:11:06.003547    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f9a093a_89b7_48ae_9123_d5b87aa9dd27.slice/crio-d062c45a8f9094d411d28e0ea5f6dc93f4664d6d9ab8585bb1a496dbe520eb27 WatchSource:0}: Error finding container d062c45a8f9094d411d28e0ea5f6dc93f4664d6d9ab8585bb1a496dbe520eb27: Status 404 returned error can't find the container with id d062c45a8f9094d411d28e0ea5f6dc93f4664d6d9ab8585bb1a496dbe520eb27
Oct 31 13:11:06.694195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:06.694153    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" event={"ID":"4f9a093a-89b7-48ae-9123-d5b87aa9dd27","Type":"ContainerStarted","Data":"4f001f00f4e004b942737c7ccc3b75935ab11d6bbe1f3f2ff9fc1b69bbfcdb89"}
Oct 31 13:11:06.694578 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:06.694203    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" event={"ID":"4f9a093a-89b7-48ae-9123-d5b87aa9dd27","Type":"ContainerStarted","Data":"d062c45a8f9094d411d28e0ea5f6dc93f4664d6d9ab8585bb1a496dbe520eb27"}
Oct 31 13:11:06.705157 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:06.705122    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:06.716416 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:06.716364    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podStartSLOduration=1.7163478269999999 podStartE2EDuration="1.716347827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:06.715692752 +0000 UTC m=+646.131634603" watchObservedRunningTime="2026-10-31 13:11:06.716347827 +0000 UTC m=+646.132289668"
Oct 31 13:11:07.227148 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:07.227110    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:11:07.240093 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:07.240065    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:11:07.711048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:07.711015    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:11:12.198731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.198682    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"]
Oct 31 13:11:12.202907 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.202872    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.209946 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.209906    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f9e24f\"/\"premium-w5-f9e24f-kserve-self-signed-certs\""
Oct 31 13:11:12.216496 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.216466    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"]
Oct 31 13:11:12.284361 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.284318    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-dshm\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.284361 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.284363    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln4nl\" (UniqueName: \"kubernetes.io/projected/d366feca-7052-4618-8a80-b929afe65e16-kube-api-access-ln4nl\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.284627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.284391    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-tmp-dir\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.284627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.284539    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d366feca-7052-4618-8a80-b929afe65e16-tls-certs\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.284627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.284606    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-model-cache\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.284819 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.284643    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-home\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.385653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.385610    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-home\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.385903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.385674    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-dshm\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.385903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.385699    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln4nl\" (UniqueName: \"kubernetes.io/projected/d366feca-7052-4618-8a80-b929afe65e16-kube-api-access-ln4nl\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.385903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.385772    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-tmp-dir\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.385903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.385825    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d366feca-7052-4618-8a80-b929afe65e16-tls-certs\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.385903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.385887    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-model-cache\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.386199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.386174    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-home\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.386255 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.386191    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-tmp-dir\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.386309 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.386274    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-model-cache\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.388194 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.388170    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-dshm\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.388354 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.388337    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d366feca-7052-4618-8a80-b929afe65e16-tls-certs\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.400261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.400234    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln4nl\" (UniqueName: \"kubernetes.io/projected/d366feca-7052-4618-8a80-b929afe65e16-kube-api-access-ln4nl\") pod \"premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") " pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.517541 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.517446    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:12.800426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:12.800394    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"]
Oct 31 13:11:12.802375 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:11:12.802350    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd366feca_7052_4618_8a80_b929afe65e16.slice/crio-2392824da8406056f27fe2a19a60452ee0ec06fcd0735119cb240fc672719982 WatchSource:0}: Error finding container 2392824da8406056f27fe2a19a60452ee0ec06fcd0735119cb240fc672719982: Status 404 returned error can't find the container with id 2392824da8406056f27fe2a19a60452ee0ec06fcd0735119cb240fc672719982
Oct 31 13:11:13.736320 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:13.736278    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" event={"ID":"d366feca-7052-4618-8a80-b929afe65e16","Type":"ContainerStarted","Data":"699973dca7537958c1cf523e6274bde1aa94bccac02682d9644f47942ceba0a7"}
Oct 31 13:11:13.736757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:13.736329    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" event={"ID":"d366feca-7052-4618-8a80-b929afe65e16","Type":"ContainerStarted","Data":"2392824da8406056f27fe2a19a60452ee0ec06fcd0735119cb240fc672719982"}
Oct 31 13:11:13.757674 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:13.757593    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podStartSLOduration=1.757577987 podStartE2EDuration="1.757577987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:13.756410898 +0000 UTC m=+653.172352742" watchObservedRunningTime="2026-10-31 13:11:13.757577987 +0000 UTC m=+653.173519827"
Oct 31 13:11:14.518465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:14.518422    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:15.705662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:15.705624    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:15.718397 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:15.718369    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:15.758375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:15.758347    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:11:16.065341 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.065258    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"]
Oct 31 13:11:16.069144 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.069121    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.086123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.086096    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"premium-w4-426787-kserve-self-signed-certs\""
Oct 31 13:11:16.091643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.091614    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"]
Oct 31 13:11:16.130449 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.130418    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-dshm\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.130449 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.130453    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tmp-dir\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.130767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.130483    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svd72\" (UniqueName: \"kubernetes.io/projected/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-kube-api-access-svd72\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.130767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.130655    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-home\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.130889 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.130814    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tls-certs\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.130889 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.130852    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-model-cache\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.237819 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.237782    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-dshm\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.238084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.238065    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tmp-dir\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.238230 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.238214    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svd72\" (UniqueName: \"kubernetes.io/projected/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-kube-api-access-svd72\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.238362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.238348    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-home\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.238520 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.238506    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tls-certs\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.238617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.238605    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-model-cache\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.239276 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.239253    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-model-cache\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.243953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.241017    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-home\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.243953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.243008    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tmp-dir\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.244915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.244880    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-dshm\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.245376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.245357    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tls-certs\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.252696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.252671    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svd72\" (UniqueName: \"kubernetes.io/projected/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-kube-api-access-svd72\") pod \"premium-w4-426787-kserve-55c99d9bb6-w8969\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") " pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.381984 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.381870    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:16.667508 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.667479    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"]
Oct 31 13:11:16.669987 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:11:16.669959    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23a61fd1_6bc0_4ea5_a1ad_a5cc957a76b0.slice/crio-19efd715d3ed78d738db8a91cf03ca033a1d8a98c5db7e1c04e06a0f73fafb55 WatchSource:0}: Error finding container 19efd715d3ed78d738db8a91cf03ca033a1d8a98c5db7e1c04e06a0f73fafb55: Status 404 returned error can't find the container with id 19efd715d3ed78d738db8a91cf03ca033a1d8a98c5db7e1c04e06a0f73fafb55
Oct 31 13:11:16.761152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:16.756909    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" event={"ID":"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0","Type":"ContainerStarted","Data":"19efd715d3ed78d738db8a91cf03ca033a1d8a98c5db7e1c04e06a0f73fafb55"}
Oct 31 13:11:17.764780 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:17.764737    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" event={"ID":"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0","Type":"ContainerStarted","Data":"bb1dfe2236ab043b0d2caca484785e4827076bbe7ef29e92ed6fce1b81486f14"}
Oct 31 13:11:17.787849 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:17.787794    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podStartSLOduration=1.787775659 podStartE2EDuration="1.787775659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:17.785435741 +0000 UTC m=+657.201377580" watchObservedRunningTime="2026-10-31 13:11:17.787775659 +0000 UTC m=+657.203717499"
Oct 31 13:11:18.247569 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:11:18.247531    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:11:18.382639 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:18.382606    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:22.517628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:22.517582    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:22.530092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:22.530060    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:22.806211 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:22.806109    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:11:26.382631 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:26.382590    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:26.396365 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:26.396337    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:26.825013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:26.824986    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:11:33.695306 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.695274    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"]
Oct 31 13:11:33.699778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.699751    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.705962 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.705941    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-79c6ff\"/\"kube-root-ca.crt\""
Oct 31 13:11:33.706119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.706103    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-79c6ff\"/\"openshift-service-ca.crt\""
Oct 31 13:11:33.706119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.706111    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-79c6ff\"/\"default-dockercfg-rkkdq\""
Oct 31 13:11:33.706221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.706204    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-79c6ff\"/\"premium-w6-79c6ff-kserve-self-signed-certs\""
Oct 31 13:11:33.711472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.711446    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"]
Oct 31 13:11:33.829483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.829439    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-dshm\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.831195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.831162    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65cqb\" (UniqueName: \"kubernetes.io/projected/1877b8bc-92f7-41c4-a6d5-88ed2b550001-kube-api-access-65cqb\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.831602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.831582    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tls-certs\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.831880 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.831851    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-home\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.832028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.832011    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-model-cache\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.832200 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.832181    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tmp-dir\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.933903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.933860    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-model-cache\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.934099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.933925    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tmp-dir\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.934099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.933991    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-dshm\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.934099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.934025    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65cqb\" (UniqueName: \"kubernetes.io/projected/1877b8bc-92f7-41c4-a6d5-88ed2b550001-kube-api-access-65cqb\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.934313 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.934158    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tls-certs\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.934313 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.934216    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-home\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.934420 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.934360    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-model-cache\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.934420 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.934402    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tmp-dir\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.934641 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.934622    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-home\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.936564 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.936544    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-dshm\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.937455 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.937378    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tls-certs\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:33.952869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:33.952796    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65cqb\" (UniqueName: \"kubernetes.io/projected/1877b8bc-92f7-41c4-a6d5-88ed2b550001-kube-api-access-65cqb\") pod \"premium-w6-79c6ff-kserve-9476bfd5f-lbmh6\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") " pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:34.017873 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:34.017836    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:34.299898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:34.299876    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"]
Oct 31 13:11:34.302110 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:11:34.302076    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1877b8bc_92f7_41c4_a6d5_88ed2b550001.slice/crio-5bf4cf4c586a5f9751ee376ae56b9ab7a8ea065e50351d05956d68faceda8d55 WatchSource:0}: Error finding container 5bf4cf4c586a5f9751ee376ae56b9ab7a8ea065e50351d05956d68faceda8d55: Status 404 returned error can't find the container with id 5bf4cf4c586a5f9751ee376ae56b9ab7a8ea065e50351d05956d68faceda8d55
Oct 31 13:11:34.863292 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:34.863255    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" event={"ID":"1877b8bc-92f7-41c4-a6d5-88ed2b550001","Type":"ContainerStarted","Data":"5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08"}
Oct 31 13:11:34.863292 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:34.863297    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" event={"ID":"1877b8bc-92f7-41c4-a6d5-88ed2b550001","Type":"ContainerStarted","Data":"5bf4cf4c586a5f9751ee376ae56b9ab7a8ea065e50351d05956d68faceda8d55"}
Oct 31 13:11:34.884766 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:34.884686    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podStartSLOduration=1.884670264 podStartE2EDuration="1.884670264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:34.882753236 +0000 UTC m=+674.298695078" watchObservedRunningTime="2026-10-31 13:11:34.884670264 +0000 UTC m=+674.300612110"
Oct 31 13:11:35.021832 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:35.021802    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:36.329569 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.329534    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"]
Oct 31 13:11:36.333153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.333128    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.340536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.340501    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-f9e24f-openshift-default-dockercfg-xqbc5\""
Oct 31 13:11:36.347145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.347106    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"]
Oct 31 13:11:36.460330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.460278    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.460330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.460335    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.460557 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.460503    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321518c6-acce-489e-9cf5-67a29bafc5ad-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.460615 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.460552    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nzjt\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-kube-api-access-9nzjt\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.460659 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.460620    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.460727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.460661    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.460727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.460686    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.460798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.460750    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.460798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.460793    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.561991 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.561953    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321518c6-acce-489e-9cf5-67a29bafc5ad-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562017    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nzjt\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-kube-api-access-9nzjt\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562065    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562128    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562166    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562416 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562201    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562416 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562237    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562416 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562281    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562416 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562312    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562651 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562559    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562686    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562805 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562772    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562844    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321518c6-acce-489e-9cf5-67a29bafc5ad-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.562921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.562898    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.564945 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.564910    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.565283 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.565261    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.576030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.576005    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.576140 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.576118    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nzjt\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-kube-api-access-9nzjt\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.651586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.651509    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:36.937817 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.937783    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"]
Oct 31 13:11:36.940818 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:11:36.940791    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321518c6_acce_489e_9cf5_67a29bafc5ad.slice/crio-290b481ab15ac6bac64e8eac2f16ae1a6dbbc5e9a5f4278f517d7146dc4b7394 WatchSource:0}: Error finding container 290b481ab15ac6bac64e8eac2f16ae1a6dbbc5e9a5f4278f517d7146dc4b7394: Status 404 returned error can't find the container with id 290b481ab15ac6bac64e8eac2f16ae1a6dbbc5e9a5f4278f517d7146dc4b7394
Oct 31 13:11:36.943565 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.943528    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:36.943642 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.943611    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:36.943682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:36.943651    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:37.883002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:37.882959    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" event={"ID":"321518c6-acce-489e-9cf5-67a29bafc5ad","Type":"ContainerStarted","Data":"a572f7af851bbb136ad598d533d7f2f8972207d9d54097132cb076a471a639ac"}
Oct 31 13:11:37.883461 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:37.883038    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" event={"ID":"321518c6-acce-489e-9cf5-67a29bafc5ad","Type":"ContainerStarted","Data":"290b481ab15ac6bac64e8eac2f16ae1a6dbbc5e9a5f4278f517d7146dc4b7394"}
Oct 31 13:11:38.652730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:38.652093    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:38.661811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:38.661087    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 13:11:38.661811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:38.661152    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 13:11:39.654727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:39.654103    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 13:11:39.654727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:39.654165    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 13:11:40.652153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:40.652117    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 13:11:40.654679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:40.654646    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 13:11:41.658761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:41.658659    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 13:11:41.659283 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:41.658769    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 13:11:42.657741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:42.655235    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 13:11:42.657741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:42.655296    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 13:11:43.651912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:43.651873    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 13:11:43.652470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:43.651935    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 13:11:44.019140 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:44.019098    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:44.039067 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:44.039036    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:44.063993 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:44.063929    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podStartSLOduration=8.063909199 podStartE2EDuration="8.063909199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:37.912547788 +0000 UTC m=+677.328489633" watchObservedRunningTime="2026-10-31 13:11:44.063909199 +0000 UTC m=+683.479851041"
Oct 31 13:11:44.934632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:44.934597    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:11:44.938215 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:44.938188    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:44.938503 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:44.938484    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:44.939246 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:44.939231    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:11:48.246465 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:11:48.246426    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:11:51.472223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.472190    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"]
Oct 31 13:11:51.477080 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.477058    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.481792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.481772    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-426787-openshift-default-dockercfg-7cnrw\""
Oct 31 13:11:51.490244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.490219    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"]
Oct 31 13:11:51.560446 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.560413    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"]
Oct 31 13:11:51.562789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.562754    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b889c3d9-985e-45d9-941e-539936378181-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.562895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.562803    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.562954 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.562930    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b889c3d9-985e-45d9-941e-539936378181-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.562990 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.562967    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjb24\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-kube-api-access-jjb24\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.563045 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.563034    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.563080 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.563061    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.563146 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.563135    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.563181 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.563164    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.563214 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.563182    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.563987 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.563974    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.582013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.581981    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"]
Oct 31 13:11:51.582163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.582054    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2a262a-openshift-default-dockercfg-bjmvk\""
Oct 31 13:11:51.663844 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.663810    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.664013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.663850    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.663881    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.663949    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.664013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.663987    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2b6j\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-kube-api-access-l2b6j\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.664173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664051    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b889c3d9-985e-45d9-941e-539936378181-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664129    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664257 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664179    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.664257 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664197    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.664257 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664220    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.664413 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664284    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664413 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664296    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664413 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664325    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b889c3d9-985e-45d9-941e-539936378181-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664413 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664363    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjb24\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-kube-api-access-jjb24\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664413 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664396    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.664667 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664464    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664667 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664497    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.664667 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664525    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.664667 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664557    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664667 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664611    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664981 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664759    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b889c3d9-985e-45d9-941e-539936378181-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664981 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664772    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.664981 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.664951    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.666560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.666538    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.666998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.666981    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b889c3d9-985e-45d9-941e-539936378181-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.677925 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.677900    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.678406 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.678383    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjb24\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-kube-api-access-jjb24\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.765697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.765606    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.765697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.765676    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.765975 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.765744    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2b6j\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-kube-api-access-l2b6j\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.766150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.766118    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.766344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.766167    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.766344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.766213    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.766344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.766267    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.766682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.766412    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.766682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.766449    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.766682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.766503    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.766920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.766756    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.766920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.766875    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.767137 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.767113    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.767235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.767177    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.768476 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.768453    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.768806 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.768786    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.779611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.779579    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.779765 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.779657    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2b6j\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-kube-api-access-l2b6j\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.790892 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.790866    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:51.878157 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.878125    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:51.926081 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.926042    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"]
Oct 31 13:11:51.930590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.930564    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:51.936083 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.936037    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7bf66c-openshift-default-dockercfg-htxtf\""
Oct 31 13:11:51.947021 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.946993    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"]
Oct 31 13:11:51.968210 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.968178    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:51.968369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.968294    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17041e62-9718-42d3-9481-30851a9c7db0-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:51.968369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.968363    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:51.968461 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.968437    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:51.968509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.968472    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g94kw\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-kube-api-access-g94kw\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:51.968509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.968498    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:51.968590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.968550    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17041e62-9718-42d3-9481-30851a9c7db0-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:51.968668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.968650    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:51.968747 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:51.968736    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069380 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069290    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069380 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069355    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069395    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069452    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17041e62-9718-42d3-9481-30851a9c7db0-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069532    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069574    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069603    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g94kw\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-kube-api-access-g94kw\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069639    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069668    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17041e62-9718-42d3-9481-30851a9c7db0-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069827    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.069927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.069897    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.070125 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.070061    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.070164 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.070142    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.070678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.070648    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17041e62-9718-42d3-9481-30851a9c7db0-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.072633 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.072613    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.072793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.072776    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17041e62-9718-42d3-9481-30851a9c7db0-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.073740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.073720    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"]
Oct 31 13:11:52.075387 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:11:52.075366    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb889c3d9_985e_45d9_941e_539936378181.slice/crio-42dd12a775c3811e90bbd4eabd6245e69f0009a93362ba37372934cd79dc172f WatchSource:0}: Error finding container 42dd12a775c3811e90bbd4eabd6245e69f0009a93362ba37372934cd79dc172f: Status 404 returned error can't find the container with id 42dd12a775c3811e90bbd4eabd6245e69f0009a93362ba37372934cd79dc172f
Oct 31 13:11:52.077972 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.077940    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:52.078065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.078015    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:52.078065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.078060    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:52.082472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.082441    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.082609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.082591    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g94kw\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-kube-api-access-g94kw\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.164215 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.164190    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"]
Oct 31 13:11:52.167394 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:11:52.167367    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb5a93b_7e4d_4eab_a4ea_fceaa2a5186b.slice/crio-5293753b1ee3b3e4438df575535bd3fdccb75106d07d17f1087e9611fb7aacdb WatchSource:0}: Error finding container 5293753b1ee3b3e4438df575535bd3fdccb75106d07d17f1087e9611fb7aacdb: Status 404 returned error can't find the container with id 5293753b1ee3b3e4438df575535bd3fdccb75106d07d17f1087e9611fb7aacdb
Oct 31 13:11:52.170343 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.170310    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:52.170434 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.170376    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:52.170434 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.170404    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:52.257400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.257373    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:52.553905 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.553875    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"]
Oct 31 13:11:52.558196 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:11:52.558167    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17041e62_9718_42d3_9481_30851a9c7db0.slice/crio-8f17848852bee00e4aff29b01605ff843c29a5d9c29c2bdcf898ca1cd7557f88 WatchSource:0}: Error finding container 8f17848852bee00e4aff29b01605ff843c29a5d9c29c2bdcf898ca1cd7557f88: Status 404 returned error can't find the container with id 8f17848852bee00e4aff29b01605ff843c29a5d9c29c2bdcf898ca1cd7557f88
Oct 31 13:11:52.561508 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.561477    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:52.561614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.561561    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:52.561687 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.561610    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:11:52.998909 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.998765    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" event={"ID":"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b","Type":"ContainerStarted","Data":"40c7478010510aadaf3083f9dec1e55926866b66cdaffac4949181502e443713"}
Oct 31 13:11:52.998909 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:52.998839    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" event={"ID":"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b","Type":"ContainerStarted","Data":"5293753b1ee3b3e4438df575535bd3fdccb75106d07d17f1087e9611fb7aacdb"}
Oct 31 13:11:53.000970 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.000942    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" event={"ID":"b889c3d9-985e-45d9-941e-539936378181","Type":"ContainerStarted","Data":"f067cb8e433ecf1b49d85f38799fa2964bed65959a8c234f03beb144a744df1f"}
Oct 31 13:11:53.001104 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.001008    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" event={"ID":"b889c3d9-985e-45d9-941e-539936378181","Type":"ContainerStarted","Data":"42dd12a775c3811e90bbd4eabd6245e69f0009a93362ba37372934cd79dc172f"}
Oct 31 13:11:53.002823 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.002800    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" event={"ID":"17041e62-9718-42d3-9481-30851a9c7db0","Type":"ContainerStarted","Data":"932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37"}
Oct 31 13:11:53.002899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.002840    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" event={"ID":"17041e62-9718-42d3-9481-30851a9c7db0","Type":"ContainerStarted","Data":"8f17848852bee00e4aff29b01605ff843c29a5d9c29c2bdcf898ca1cd7557f88"}
Oct 31 13:11:53.027533 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.027487    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" podStartSLOduration=2.027472265 podStartE2EDuration="2.027472265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:53.025869368 +0000 UTC m=+692.441811212" watchObservedRunningTime="2026-10-31 13:11:53.027472265 +0000 UTC m=+692.443414105"
Oct 31 13:11:53.099103 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.099040    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podStartSLOduration=2.099019954 podStartE2EDuration="2.099019954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:53.05450358 +0000 UTC m=+692.470445421" watchObservedRunningTime="2026-10-31 13:11:53.099019954 +0000 UTC m=+692.514961796"
Oct 31 13:11:53.100413 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.100367    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podStartSLOduration=2.100354072 podStartE2EDuration="2.100354072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:11:53.095889679 +0000 UTC m=+692.511831520" watchObservedRunningTime="2026-10-31 13:11:53.100354072 +0000 UTC m=+692.516295917"
Oct 31 13:11:53.259790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.259633    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:11:53.259976 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.259891    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 13:11:53.259976 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.259955    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 13:11:53.791370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.791316    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:53.793091 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.793062    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 31 13:11:53.793239 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.793154    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 31 13:11:53.881729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.879090    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:11:53.881729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.881172    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 13:11:53.881729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:53.881229    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 13:11:54.261737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:54.260822    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 13:11:54.261737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:54.260882    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 13:11:54.794726 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:54.794119    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 31 13:11:54.794726 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:54.794184    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 31 13:11:54.886407 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:54.879174    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 13:11:54.886407 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:54.879240    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 13:11:55.258731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:55.258119    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 13:11:55.258731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:55.258186    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 13:11:55.793522 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:55.793486    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 31 13:11:55.793863 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:55.793832    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 31 13:11:55.881731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:55.879143    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 13:11:55.881731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:55.879206    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 13:11:56.263175 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:56.263137    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 13:11:56.263472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:56.263445    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 13:11:56.804729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:56.796854    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 31 13:11:56.804729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:56.796927    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 31 13:11:56.884730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:56.884069    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 13:11:56.884730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:56.884131    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 13:11:57.260250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.260205    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 13:11:57.260543 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.260510    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 13:11:57.645026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.644919    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:11:57.705198 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.704800    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:11:57.754853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.754800    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:11:57.794808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.794774    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 31 13:11:57.795080 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.795057    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 31 13:11:57.819857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.819801    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:11:57.884660 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.884624    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 13:11:57.884964 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.884934    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 13:11:57.939140 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:57.939085    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:11:58.040734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:58.039829    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:11:58.230800 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:58.230554    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:11:58.258241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:58.258197    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 13:11:58.258563 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:58.258525    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 13:11:58.791521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:58.791484    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Oct 31 13:11:58.791685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:58.791556    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Oct 31 13:11:58.879059 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:58.879025    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Oct 31 13:11:58.879274 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:58.879086    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Oct 31 13:11:59.258179 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:59.258138    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 13:11:59.258837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:59.258790    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 13:11:59.796349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:59.796315    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:59.796642 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:59.796623    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:11:59.797390 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:11:59.797366    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:12:00.258892 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:00.258853    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 13:12:00.259547 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:00.259512    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 13:12:00.337929 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:00.337898    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:12:00.338432 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:00.338354    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:12:00.340259 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:00.340196    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:12:01.262849 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:01.262820    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:12:01.263250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:01.263112    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:12:01.263794 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:01.263774    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:12:09.292655 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.292615    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"]
Oct 31 13:12:09.296777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.296755    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.305669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.305476    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:12:09.305669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.305515    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:09.305669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.305587    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:12:09.305669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.305635    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:12:09.307057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.306989    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"]
Oct 31 13:12:09.457274 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.457226    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a1fdcc8-nwd8v\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.457448 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.457365    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw46r\" (UniqueName: \"kubernetes.io/projected/1804f562-6f27-4a45-93ce-887da37f1590-kube-api-access-pw46r\") pod \"maas-api-revoke-sub-3a1fdcc8-nwd8v\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.457448 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.457422    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a1fdcc8-nwd8v\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.558121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.558024    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw46r\" (UniqueName: \"kubernetes.io/projected/1804f562-6f27-4a45-93ce-887da37f1590-kube-api-access-pw46r\") pod \"maas-api-revoke-sub-3a1fdcc8-nwd8v\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.558121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.558102    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a1fdcc8-nwd8v\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.558364 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.558182    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a1fdcc8-nwd8v\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.558811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.558784    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a1fdcc8-nwd8v\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.560644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.560622    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a1fdcc8-nwd8v\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.571956 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.571925    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw46r\" (UniqueName: \"kubernetes.io/projected/1804f562-6f27-4a45-93ce-887da37f1590-kube-api-access-pw46r\") pod \"maas-api-revoke-sub-3a1fdcc8-nwd8v\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.613568 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.613528    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:09.897128 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.897102    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"]
Oct 31 13:12:09.899140 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:09.899099    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1804f562_6f27_4a45_93ce_887da37f1590.slice/crio-ec1ed9b50897985d807215ac3d28a56e12808c70c1d2183a0d4c875ad0d8eb5d WatchSource:0}: Error finding container ec1ed9b50897985d807215ac3d28a56e12808c70c1d2183a0d4c875ad0d8eb5d: Status 404 returned error can't find the container with id ec1ed9b50897985d807215ac3d28a56e12808c70c1d2183a0d4c875ad0d8eb5d
Oct 31 13:12:09.901216 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:09.901193    5787 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 13:12:10.292944 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.292894    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v" event={"ID":"1804f562-6f27-4a45-93ce-887da37f1590","Type":"ContainerStarted","Data":"ec1ed9b50897985d807215ac3d28a56e12808c70c1d2183a0d4c875ad0d8eb5d"}
Oct 31 13:12:10.427895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.427856    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"]
Oct 31 13:12:10.432943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.432914    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.441392 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.441362    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"]
Oct 31 13:12:10.484440 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.484404    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb10808-c52a-4e25-b152-50f946432b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ac55d3-vw2zd\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.484598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.484450    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb10808-c52a-4e25-b152-50f946432b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ac55d3-vw2zd\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.484598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.484576    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2wct\" (UniqueName: \"kubernetes.io/projected/efb10808-c52a-4e25-b152-50f946432b19-kube-api-access-p2wct\") pod \"maas-api-revoke-sub-e5ac55d3-vw2zd\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.587168 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.586212    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb10808-c52a-4e25-b152-50f946432b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ac55d3-vw2zd\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.587168 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.586333    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2wct\" (UniqueName: \"kubernetes.io/projected/efb10808-c52a-4e25-b152-50f946432b19-kube-api-access-p2wct\") pod \"maas-api-revoke-sub-e5ac55d3-vw2zd\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.587168 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.586514    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb10808-c52a-4e25-b152-50f946432b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ac55d3-vw2zd\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.588676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.588314    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb10808-c52a-4e25-b152-50f946432b19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5ac55d3-vw2zd\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.593493 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.591389    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb10808-c52a-4e25-b152-50f946432b19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5ac55d3-vw2zd\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.601882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.601850    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2wct\" (UniqueName: \"kubernetes.io/projected/efb10808-c52a-4e25-b152-50f946432b19-kube-api-access-p2wct\") pod \"maas-api-revoke-sub-e5ac55d3-vw2zd\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:10.749344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:10.749311    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:11.033662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:11.033627    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"]
Oct 31 13:12:11.039734 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:11.035655    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb10808_c52a_4e25_b152_50f946432b19.slice/crio-fd050948e7a3f1377e25989c64f70d848a75248b42bdaeaf79348fafda3d1930 WatchSource:0}: Error finding container fd050948e7a3f1377e25989c64f70d848a75248b42bdaeaf79348fafda3d1930: Status 404 returned error can't find the container with id fd050948e7a3f1377e25989c64f70d848a75248b42bdaeaf79348fafda3d1930
Oct 31 13:12:11.317806 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:11.313057    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd" event={"ID":"efb10808-c52a-4e25-b152-50f946432b19","Type":"ContainerStarted","Data":"fd050948e7a3f1377e25989c64f70d848a75248b42bdaeaf79348fafda3d1930"}
Oct 31 13:12:13.326645 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:13.326539    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v" event={"ID":"1804f562-6f27-4a45-93ce-887da37f1590","Type":"ContainerStarted","Data":"f8d45d7e814ecc32ee0ae70dedea3d47c13668b48fd2eae5c22d6ff6b8c1f210"}
Oct 31 13:12:13.328030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:13.328004    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd" event={"ID":"efb10808-c52a-4e25-b152-50f946432b19","Type":"ContainerStarted","Data":"27afeebd1dd5446f0c08a437493f8e8084ca0aa010ba4b54773d34fcdaa8e92a"}
Oct 31 13:12:13.348337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:13.348285    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v" podStartSLOduration=1.551597772 podStartE2EDuration="4.348271901s" totalImagesPullingTime="2.796674129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:13.347845326 +0000 UTC m=+712.763787181" watchObservedRunningTime="2026-10-31 13:12:13.348271901 +0000 UTC m=+712.764213739"
Oct 31 13:12:13.370618 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:13.370565    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd" podStartSLOduration=1.714158325 podStartE2EDuration="3.370550314s" totalImagesPullingTime="1.656391989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:13.369337093 +0000 UTC m=+712.785278934" watchObservedRunningTime="2026-10-31 13:12:13.370550314 +0000 UTC m=+712.786492153"
Oct 31 13:12:14.335432 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:14.335384    5787 generic.go:356] "Generic (PLEG): container finished" podID="efb10808-c52a-4e25-b152-50f946432b19" containerID="27afeebd1dd5446f0c08a437493f8e8084ca0aa010ba4b54773d34fcdaa8e92a" exitCode=0
Oct 31 13:12:14.335952 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:14.335468    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd" event={"ID":"efb10808-c52a-4e25-b152-50f946432b19","Type":"ContainerDied","Data":"27afeebd1dd5446f0c08a437493f8e8084ca0aa010ba4b54773d34fcdaa8e92a"}
Oct 31 13:12:14.337223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:14.337197    5787 generic.go:356] "Generic (PLEG): container finished" podID="1804f562-6f27-4a45-93ce-887da37f1590" containerID="f8d45d7e814ecc32ee0ae70dedea3d47c13668b48fd2eae5c22d6ff6b8c1f210" exitCode=0
Oct 31 13:12:14.337341 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:14.337235    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v" event={"ID":"1804f562-6f27-4a45-93ce-887da37f1590","Type":"ContainerDied","Data":"f8d45d7e814ecc32ee0ae70dedea3d47c13668b48fd2eae5c22d6ff6b8c1f210"}
Oct 31 13:12:15.521266 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.521236    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:15.524099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.524078    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:15.551895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.551864    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-openshift-ca\") pod \"1804f562-6f27-4a45-93ce-887da37f1590\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") "
Oct 31 13:12:15.552096 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.551933    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb10808-c52a-4e25-b152-50f946432b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb10808-c52a-4e25-b152-50f946432b19-maas-api-openshift-ca\") pod \"efb10808-c52a-4e25-b152-50f946432b19\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") "
Oct 31 13:12:15.552096 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.552079    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-xks-ca\") pod \"1804f562-6f27-4a45-93ce-887da37f1590\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") "
Oct 31 13:12:15.552271 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.552139    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1804f562-6f27-4a45-93ce-887da37f1590-kube-api-access-pw46r\" (UniqueName: \"kubernetes.io/projected/1804f562-6f27-4a45-93ce-887da37f1590-kube-api-access-pw46r\") pod \"1804f562-6f27-4a45-93ce-887da37f1590\" (UID: \"1804f562-6f27-4a45-93ce-887da37f1590\") "
Oct 31 13:12:15.552271 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.552172    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efb10808-c52a-4e25-b152-50f946432b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb10808-c52a-4e25-b152-50f946432b19-maas-api-xks-ca\") pod \"efb10808-c52a-4e25-b152-50f946432b19\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") "
Oct 31 13:12:15.552271 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.552245    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb10808-c52a-4e25-b152-50f946432b19-kube-api-access-p2wct\" (UniqueName: \"kubernetes.io/projected/efb10808-c52a-4e25-b152-50f946432b19-kube-api-access-p2wct\") pod \"efb10808-c52a-4e25-b152-50f946432b19\" (UID: \"efb10808-c52a-4e25-b152-50f946432b19\") "
Oct 31 13:12:15.552418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.552280    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-openshift-ca" pod "1804f562-6f27-4a45-93ce-887da37f1590" (UID: "1804f562-6f27-4a45-93ce-887da37f1590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:15.552418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.552356    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb10808-c52a-4e25-b152-50f946432b19-maas-api-openshift-ca" pod "efb10808-c52a-4e25-b152-50f946432b19" (UID: "efb10808-c52a-4e25-b152-50f946432b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:15.553359 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.553261    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:15.553481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.553362    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efb10808-c52a-4e25-b152-50f946432b19-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:15.555312 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.555265    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-xks-ca" pod "1804f562-6f27-4a45-93ce-887da37f1590" (UID: "1804f562-6f27-4a45-93ce-887da37f1590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:15.555312 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.555260    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efb10808-c52a-4e25-b152-50f946432b19-maas-api-xks-ca" pod "efb10808-c52a-4e25-b152-50f946432b19" (UID: "efb10808-c52a-4e25-b152-50f946432b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:15.555312 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.555304    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1804f562-6f27-4a45-93ce-887da37f1590-kube-api-access-pw46r" pod "1804f562-6f27-4a45-93ce-887da37f1590" (UID: "1804f562-6f27-4a45-93ce-887da37f1590"). InnerVolumeSpecName "kube-api-access-pw46r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:15.555535 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.555335    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb10808-c52a-4e25-b152-50f946432b19-kube-api-access-p2wct" pod "efb10808-c52a-4e25-b152-50f946432b19" (UID: "efb10808-c52a-4e25-b152-50f946432b19"). InnerVolumeSpecName "kube-api-access-p2wct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:15.654611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.654522    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1804f562-6f27-4a45-93ce-887da37f1590-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:15.654611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.654550    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw46r\" (UniqueName: \"kubernetes.io/projected/1804f562-6f27-4a45-93ce-887da37f1590-kube-api-access-pw46r\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:15.654611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.654562    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efb10808-c52a-4e25-b152-50f946432b19-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:15.654611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:15.654571    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2wct\" (UniqueName: \"kubernetes.io/projected/efb10808-c52a-4e25-b152-50f946432b19-kube-api-access-p2wct\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:16.353727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:16.353369    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v" event={"ID":"1804f562-6f27-4a45-93ce-887da37f1590","Type":"ContainerDied","Data":"ec1ed9b50897985d807215ac3d28a56e12808c70c1d2183a0d4c875ad0d8eb5d"}
Oct 31 13:12:16.353727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:16.353439    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec1ed9b50897985d807215ac3d28a56e12808c70c1d2183a0d4c875ad0d8eb5d"
Oct 31 13:12:16.353727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:16.353525    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"
Oct 31 13:12:16.374729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:16.363684    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd" event={"ID":"efb10808-c52a-4e25-b152-50f946432b19","Type":"ContainerDied","Data":"fd050948e7a3f1377e25989c64f70d848a75248b42bdaeaf79348fafda3d1930"}
Oct 31 13:12:16.374729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:16.363770    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd050948e7a3f1377e25989c64f70d848a75248b42bdaeaf79348fafda3d1930"
Oct 31 13:12:16.374729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:16.363844    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"
Oct 31 13:12:17.203310 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.203279    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"]
Oct 31 13:12:17.203739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.203730    5787 cpu_manager.go:414] "containerMap: removing container" podUID="1804f562-6f27-4a45-93ce-887da37f1590" containerName="revoke-keys"
Oct 31 13:12:17.203796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.203741    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1804f562-6f27-4a45-93ce-887da37f1590" containerName="revoke-keys"
Oct 31 13:12:17.203796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.203754    5787 cpu_manager.go:414] "containerMap: removing container" podUID="efb10808-c52a-4e25-b152-50f946432b19" containerName="revoke-keys"
Oct 31 13:12:17.203796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.203759    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb10808-c52a-4e25-b152-50f946432b19" containerName="revoke-keys"
Oct 31 13:12:17.203887 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.203849    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="1804f562-6f27-4a45-93ce-887da37f1590" containerName="revoke-keys"
Oct 31 13:12:17.203887 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.203858    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb10808-c52a-4e25-b152-50f946432b19" containerName="revoke-keys"
Oct 31 13:12:17.207154 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.207137    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.215164 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.215143    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:17.216034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.216014    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:12:17.216131 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.216091    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:12:17.216131 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.216111    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:12:17.221419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.221394    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"]
Oct 31 13:12:17.283333 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.283298    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5r5\" (UniqueName: \"kubernetes.io/projected/53ac0f77-88ae-4123-9418-82502b4eb3bf-kube-api-access-6p5r5\") pod \"maas-api-revoke-sub-24bd05d4-jmjfh\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.283536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.283437    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24bd05d4-jmjfh\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.283536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.283462    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24bd05d4-jmjfh\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.392888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.392842    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5r5\" (UniqueName: \"kubernetes.io/projected/53ac0f77-88ae-4123-9418-82502b4eb3bf-kube-api-access-6p5r5\") pod \"maas-api-revoke-sub-24bd05d4-jmjfh\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.393109 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.393027    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24bd05d4-jmjfh\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.393109 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.393061    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24bd05d4-jmjfh\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.393887 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.393818    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24bd05d4-jmjfh\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.395620 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.395597    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24bd05d4-jmjfh\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.407181 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.407156    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5r5\" (UniqueName: \"kubernetes.io/projected/53ac0f77-88ae-4123-9418-82502b4eb3bf-kube-api-access-6p5r5\") pod \"maas-api-revoke-sub-24bd05d4-jmjfh\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.520868 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.520779    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:17.797500 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:17.797472    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"]
Oct 31 13:12:17.799931 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:17.799903    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ac0f77_88ae_4123_9418_82502b4eb3bf.slice/crio-5a01e2a0c0ea150f83184323dc56a35f5c858061a82dd08e1cc474e72938f7af WatchSource:0}: Error finding container 5a01e2a0c0ea150f83184323dc56a35f5c858061a82dd08e1cc474e72938f7af: Status 404 returned error can't find the container with id 5a01e2a0c0ea150f83184323dc56a35f5c858061a82dd08e1cc474e72938f7af
Oct 31 13:12:18.261388 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:12:18.261350    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:12:18.396453 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:18.396368    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh" event={"ID":"53ac0f77-88ae-4123-9418-82502b4eb3bf","Type":"ContainerStarted","Data":"c4f3ac799b76290dab93e4a69f4211c02ae4562f7b66b2a074e3fa1fd8382c00"}
Oct 31 13:12:18.396640 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:18.396464    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh" event={"ID":"53ac0f77-88ae-4123-9418-82502b4eb3bf","Type":"ContainerStarted","Data":"5a01e2a0c0ea150f83184323dc56a35f5c858061a82dd08e1cc474e72938f7af"}
Oct 31 13:12:18.420478 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:18.420424    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh" podStartSLOduration=1.420410182 podStartE2EDuration="1.420410182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:18.419073408 +0000 UTC m=+717.835015247" watchObservedRunningTime="2026-10-31 13:12:18.420410182 +0000 UTC m=+717.836352055"
Oct 31 13:12:19.403657 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:19.403623    5787 generic.go:356] "Generic (PLEG): container finished" podID="53ac0f77-88ae-4123-9418-82502b4eb3bf" containerID="c4f3ac799b76290dab93e4a69f4211c02ae4562f7b66b2a074e3fa1fd8382c00" exitCode=0
Oct 31 13:12:19.404106 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:19.403726    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh" event={"ID":"53ac0f77-88ae-4123-9418-82502b4eb3bf","Type":"ContainerDied","Data":"c4f3ac799b76290dab93e4a69f4211c02ae4562f7b66b2a074e3fa1fd8382c00"}
Oct 31 13:12:20.553284 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.553256    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:20.635074 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.635037    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ac0f77-88ae-4123-9418-82502b4eb3bf-kube-api-access-6p5r5\" (UniqueName: \"kubernetes.io/projected/53ac0f77-88ae-4123-9418-82502b4eb3bf-kube-api-access-6p5r5\") pod \"53ac0f77-88ae-4123-9418-82502b4eb3bf\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") "
Oct 31 13:12:20.635074 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.635090    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-openshift-ca\") pod \"53ac0f77-88ae-4123-9418-82502b4eb3bf\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") "
Oct 31 13:12:20.635343 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.635244    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-xks-ca\") pod \"53ac0f77-88ae-4123-9418-82502b4eb3bf\" (UID: \"53ac0f77-88ae-4123-9418-82502b4eb3bf\") "
Oct 31 13:12:20.636207 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.636136    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-openshift-ca" pod "53ac0f77-88ae-4123-9418-82502b4eb3bf" (UID: "53ac0f77-88ae-4123-9418-82502b4eb3bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:20.643614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.643575    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-xks-ca" pod "53ac0f77-88ae-4123-9418-82502b4eb3bf" (UID: "53ac0f77-88ae-4123-9418-82502b4eb3bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:20.643799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.643683    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ac0f77-88ae-4123-9418-82502b4eb3bf-kube-api-access-6p5r5" pod "53ac0f77-88ae-4123-9418-82502b4eb3bf" (UID: "53ac0f77-88ae-4123-9418-82502b4eb3bf"). InnerVolumeSpecName "kube-api-access-6p5r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:20.736351 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.736315    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:20.736351 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.736344    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5r5\" (UniqueName: \"kubernetes.io/projected/53ac0f77-88ae-4123-9418-82502b4eb3bf-kube-api-access-6p5r5\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:20.736351 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:20.736356    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53ac0f77-88ae-4123-9418-82502b4eb3bf-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:21.420479 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.420448    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"
Oct 31 13:12:21.420649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.420477    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh" event={"ID":"53ac0f77-88ae-4123-9418-82502b4eb3bf","Type":"ContainerDied","Data":"5a01e2a0c0ea150f83184323dc56a35f5c858061a82dd08e1cc474e72938f7af"}
Oct 31 13:12:21.420649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.420542    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a01e2a0c0ea150f83184323dc56a35f5c858061a82dd08e1cc474e72938f7af"
Oct 31 13:12:21.503062 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.503029    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"]
Oct 31 13:12:21.503586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.503557    5787 cpu_manager.go:414] "containerMap: removing container" podUID="53ac0f77-88ae-4123-9418-82502b4eb3bf" containerName="revoke-keys"
Oct 31 13:12:21.503586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.503581    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ac0f77-88ae-4123-9418-82502b4eb3bf" containerName="revoke-keys"
Oct 31 13:12:21.503813 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.503695    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ac0f77-88ae-4123-9418-82502b4eb3bf" containerName="revoke-keys"
Oct 31 13:12:21.507059 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.507035    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.513505 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.513481    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:12:21.513652 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.513572    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:12:21.516142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.515813    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:21.516142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.515914    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:12:21.516923 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.516890    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"]
Oct 31 13:12:21.647007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.646969    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d3f12d-fclbf\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.647007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.647007    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d3f12d-fclbf\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.647439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.647031    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzw7c\" (UniqueName: \"kubernetes.io/projected/83df7efa-30cc-4341-825e-934fa3fc5eeb-kube-api-access-vzw7c\") pod \"maas-api-revoke-sub-55d3f12d-fclbf\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.748058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.748018    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d3f12d-fclbf\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.748058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.748059    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d3f12d-fclbf\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.748332 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.748085    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzw7c\" (UniqueName: \"kubernetes.io/projected/83df7efa-30cc-4341-825e-934fa3fc5eeb-kube-api-access-vzw7c\") pod \"maas-api-revoke-sub-55d3f12d-fclbf\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.748783 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.748759    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55d3f12d-fclbf\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.750744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.750717    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55d3f12d-fclbf\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.761980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.761953    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzw7c\" (UniqueName: \"kubernetes.io/projected/83df7efa-30cc-4341-825e-934fa3fc5eeb-kube-api-access-vzw7c\") pod \"maas-api-revoke-sub-55d3f12d-fclbf\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:21.822509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:21.822475    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:22.101819 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:22.101791    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"]
Oct 31 13:12:22.103800 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:22.103771    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83df7efa_30cc_4341_825e_934fa3fc5eeb.slice/crio-c5d6c13e61b6b775e98fe84d24885ed8152637287648df20a582bc38d4ff785d WatchSource:0}: Error finding container c5d6c13e61b6b775e98fe84d24885ed8152637287648df20a582bc38d4ff785d: Status 404 returned error can't find the container with id c5d6c13e61b6b775e98fe84d24885ed8152637287648df20a582bc38d4ff785d
Oct 31 13:12:22.447260 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:22.443573    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf" event={"ID":"83df7efa-30cc-4341-825e-934fa3fc5eeb","Type":"ContainerStarted","Data":"c5d6c13e61b6b775e98fe84d24885ed8152637287648df20a582bc38d4ff785d"}
Oct 31 13:12:23.450453 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:23.450415    5787 generic.go:356] "Generic (PLEG): container finished" podID="83df7efa-30cc-4341-825e-934fa3fc5eeb" containerID="6b7ba812d80b6d8bb7a9e116afaaf9f5f523b82bf5fa7626a51ea1bd0cdd698e" exitCode=0
Oct 31 13:12:23.450904 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:23.450495    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf" event={"ID":"83df7efa-30cc-4341-825e-934fa3fc5eeb","Type":"ContainerDied","Data":"6b7ba812d80b6d8bb7a9e116afaaf9f5f523b82bf5fa7626a51ea1bd0cdd698e"}
Oct 31 13:12:24.601723 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.601686    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:24.680266 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.680230    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-openshift-ca\") pod \"83df7efa-30cc-4341-825e-934fa3fc5eeb\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") "
Oct 31 13:12:24.680470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.680279    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-xks-ca\") pod \"83df7efa-30cc-4341-825e-934fa3fc5eeb\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") "
Oct 31 13:12:24.680470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.680357    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83df7efa-30cc-4341-825e-934fa3fc5eeb-kube-api-access-vzw7c\" (UniqueName: \"kubernetes.io/projected/83df7efa-30cc-4341-825e-934fa3fc5eeb-kube-api-access-vzw7c\") pod \"83df7efa-30cc-4341-825e-934fa3fc5eeb\" (UID: \"83df7efa-30cc-4341-825e-934fa3fc5eeb\") "
Oct 31 13:12:24.680736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.680669    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-openshift-ca" pod "83df7efa-30cc-4341-825e-934fa3fc5eeb" (UID: "83df7efa-30cc-4341-825e-934fa3fc5eeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:24.680987 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.680964    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:24.682787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.682759    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-xks-ca" pod "83df7efa-30cc-4341-825e-934fa3fc5eeb" (UID: "83df7efa-30cc-4341-825e-934fa3fc5eeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:24.682875 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.682810    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83df7efa-30cc-4341-825e-934fa3fc5eeb-kube-api-access-vzw7c" pod "83df7efa-30cc-4341-825e-934fa3fc5eeb" (UID: "83df7efa-30cc-4341-825e-934fa3fc5eeb"). InnerVolumeSpecName "kube-api-access-vzw7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:24.782922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.782888    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83df7efa-30cc-4341-825e-934fa3fc5eeb-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:24.782922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:24.782915    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzw7c\" (UniqueName: \"kubernetes.io/projected/83df7efa-30cc-4341-825e-934fa3fc5eeb-kube-api-access-vzw7c\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:25.463547 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:25.463506    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf" event={"ID":"83df7efa-30cc-4341-825e-934fa3fc5eeb","Type":"ContainerDied","Data":"c5d6c13e61b6b775e98fe84d24885ed8152637287648df20a582bc38d4ff785d"}
Oct 31 13:12:25.463740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:25.463598    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5d6c13e61b6b775e98fe84d24885ed8152637287648df20a582bc38d4ff785d"
Oct 31 13:12:25.463740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:25.463518    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"
Oct 31 13:12:26.542760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.542719    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"]
Oct 31 13:12:26.543191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.543178    5787 cpu_manager.go:414] "containerMap: removing container" podUID="83df7efa-30cc-4341-825e-934fa3fc5eeb" containerName="revoke-keys"
Oct 31 13:12:26.543191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.543191    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83df7efa-30cc-4341-825e-934fa3fc5eeb" containerName="revoke-keys"
Oct 31 13:12:26.543319 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.543260    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="83df7efa-30cc-4341-825e-934fa3fc5eeb" containerName="revoke-keys"
Oct 31 13:12:26.547518 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.547500    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.557751 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.557696    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:26.558967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.558943    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:12:26.559143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.558997    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:12:26.559206 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.559012    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:12:26.559585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.559563    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"]
Oct 31 13:12:26.713431 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.713401    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37892070-9deb-4766-aa50-232e30892617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6a0499-lzmm5\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.713588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.713447    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37892070-9deb-4766-aa50-232e30892617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6a0499-lzmm5\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.713588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.713522    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5xm4\" (UniqueName: \"kubernetes.io/projected/37892070-9deb-4766-aa50-232e30892617-kube-api-access-v5xm4\") pod \"maas-api-revoke-sub-7e6a0499-lzmm5\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.814443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.814343    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37892070-9deb-4766-aa50-232e30892617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6a0499-lzmm5\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.814443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.814397    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37892070-9deb-4766-aa50-232e30892617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6a0499-lzmm5\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.814443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.814428    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5xm4\" (UniqueName: \"kubernetes.io/projected/37892070-9deb-4766-aa50-232e30892617-kube-api-access-v5xm4\") pod \"maas-api-revoke-sub-7e6a0499-lzmm5\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.815259 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.815230    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37892070-9deb-4766-aa50-232e30892617-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6a0499-lzmm5\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.817150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.817128    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37892070-9deb-4766-aa50-232e30892617-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6a0499-lzmm5\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.831409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.831379    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5xm4\" (UniqueName: \"kubernetes.io/projected/37892070-9deb-4766-aa50-232e30892617-kube-api-access-v5xm4\") pod \"maas-api-revoke-sub-7e6a0499-lzmm5\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:26.861310 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:26.861277    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:27.090893 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.090770    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"]
Oct 31 13:12:27.095792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.095770    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.108176 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.108145    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"]
Oct 31 13:12:27.142397 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.142371    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"]
Oct 31 13:12:27.144480 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:27.144452    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37892070_9deb_4766_aa50_232e30892617.slice/crio-ea0e8631ad72a59eeac118c322c459ec8a8d8032acb11b1a263beabb577d1827 WatchSource:0}: Error finding container ea0e8631ad72a59eeac118c322c459ec8a8d8032acb11b1a263beabb577d1827: Status 404 returned error can't find the container with id ea0e8631ad72a59eeac118c322c459ec8a8d8032acb11b1a263beabb577d1827
Oct 31 13:12:27.219067 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.219031    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqwwp\" (UniqueName: \"kubernetes.io/projected/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-kube-api-access-dqwwp\") pod \"maas-api-revoke-sub-a578352e-m554j\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.219234 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.219076    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a578352e-m554j\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.219280 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.219229    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a578352e-m554j\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.320550 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.320508    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqwwp\" (UniqueName: \"kubernetes.io/projected/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-kube-api-access-dqwwp\") pod \"maas-api-revoke-sub-a578352e-m554j\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.320550 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.320558    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a578352e-m554j\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.320827 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.320625    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a578352e-m554j\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.321328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.321310    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a578352e-m554j\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.323099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.323081    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a578352e-m554j\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.334592 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.334566    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqwwp\" (UniqueName: \"kubernetes.io/projected/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-kube-api-access-dqwwp\") pod \"maas-api-revoke-sub-a578352e-m554j\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.412776 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.412739    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:27.492978 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.492944    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5" event={"ID":"37892070-9deb-4766-aa50-232e30892617","Type":"ContainerStarted","Data":"185ee587f57664a2b0334cf86c1ba4dcceec3591922ac547406850c5e459a284"}
Oct 31 13:12:27.493064 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.493009    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5" event={"ID":"37892070-9deb-4766-aa50-232e30892617","Type":"ContainerStarted","Data":"ea0e8631ad72a59eeac118c322c459ec8a8d8032acb11b1a263beabb577d1827"}
Oct 31 13:12:27.520537 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.520471    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5" podStartSLOduration=1.5204488779999998 podStartE2EDuration="1.520448878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:27.516912554 +0000 UTC m=+726.932854376" watchObservedRunningTime="2026-10-31 13:12:27.520448878 +0000 UTC m=+726.936390720"
Oct 31 13:12:27.714755 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:27.714724    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"]
Oct 31 13:12:27.717223 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:27.717195    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47f6dd5_cd11_4f17_bce4_1497bb5b9769.slice/crio-dcfde739e8323c90500dab262fbc96bc8097ed58f36d5d1588fe373d26e239a4 WatchSource:0}: Error finding container dcfde739e8323c90500dab262fbc96bc8097ed58f36d5d1588fe373d26e239a4: Status 404 returned error can't find the container with id dcfde739e8323c90500dab262fbc96bc8097ed58f36d5d1588fe373d26e239a4
Oct 31 13:12:28.368781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.368544    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"]
Oct 31 13:12:28.414729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.410928    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"]
Oct 31 13:12:28.414729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.411035    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.450139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.443361    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a717fc-c64tj\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.450139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.443507    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pg8\" (UniqueName: \"kubernetes.io/projected/96c93acf-213a-4eaf-8cec-d9f895ea34aa-kube-api-access-f9pg8\") pod \"maas-api-revoke-sub-92a717fc-c64tj\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.450139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.443535    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a717fc-c64tj\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.471252 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.471221    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"]
Oct 31 13:12:28.475155 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.475132    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:28.489836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.483804    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"]
Oct 31 13:12:28.530688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.522887    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j" event={"ID":"a47f6dd5-cd11-4f17-bce4-1497bb5b9769","Type":"ContainerStarted","Data":"3d62ad61489657ed6122b2cca961ab7cd996b4ef5105c9df28c1fa60cb871890"}
Oct 31 13:12:28.530688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.522992    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j" event={"ID":"a47f6dd5-cd11-4f17-bce4-1497bb5b9769","Type":"ContainerStarted","Data":"dcfde739e8323c90500dab262fbc96bc8097ed58f36d5d1588fe373d26e239a4"}
Oct 31 13:12:28.530688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.525636    5787 generic.go:356] "Generic (PLEG): container finished" podID="37892070-9deb-4766-aa50-232e30892617" containerID="185ee587f57664a2b0334cf86c1ba4dcceec3591922ac547406850c5e459a284" exitCode=0
Oct 31 13:12:28.530688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.525844    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5" event={"ID":"37892070-9deb-4766-aa50-232e30892617","Type":"ContainerDied","Data":"185ee587f57664a2b0334cf86c1ba4dcceec3591922ac547406850c5e459a284"}
Oct 31 13:12:28.548611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.548573    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pg8\" (UniqueName: \"kubernetes.io/projected/96c93acf-213a-4eaf-8cec-d9f895ea34aa-kube-api-access-f9pg8\") pod \"maas-api-revoke-sub-92a717fc-c64tj\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.578768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.574336    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a717fc-c64tj\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.578768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.574496    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a717fc-c64tj\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.578768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.574921    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pg8\" (UniqueName: \"kubernetes.io/projected/96c93acf-213a-4eaf-8cec-d9f895ea34aa-kube-api-access-f9pg8\") pod \"maas-api-revoke-sub-92a717fc-c64tj\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.578768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.567642    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j" podStartSLOduration=1.567619154 podStartE2EDuration="1.567619154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:28.546528756 +0000 UTC m=+727.962470597" watchObservedRunningTime="2026-10-31 13:12:28.567619154 +0000 UTC m=+727.983560995"
Oct 31 13:12:28.578768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.575722    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-92a717fc-c64tj\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.586732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.583256    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-92a717fc-c64tj\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.676773 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.675794    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c1b14b-njqmc\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:28.676773 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.675886    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c1b14b-njqmc\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:28.676773 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.676015    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q67qt\" (UniqueName: \"kubernetes.io/projected/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-kube-api-access-q67qt\") pod \"maas-api-revoke-sub-09c1b14b-njqmc\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:28.738226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.738189    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:28.778912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.777896    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q67qt\" (UniqueName: \"kubernetes.io/projected/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-kube-api-access-q67qt\") pod \"maas-api-revoke-sub-09c1b14b-njqmc\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:28.778912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.778107    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c1b14b-njqmc\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:28.778912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.778190    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c1b14b-njqmc\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:28.779678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.779651    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09c1b14b-njqmc\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:28.789430 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.789023    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09c1b14b-njqmc\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:28.804293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:28.804157    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q67qt\" (UniqueName: \"kubernetes.io/projected/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-kube-api-access-q67qt\") pod \"maas-api-revoke-sub-09c1b14b-njqmc\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:29.059614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.059569    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"]
Oct 31 13:12:29.062763 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:29.062673    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96c93acf_213a_4eaf_8cec_d9f895ea34aa.slice/crio-1093c2090d17005a76b9fe001d414c9597c0c3ed2a6cb37c4b97a2767eec8d3e WatchSource:0}: Error finding container 1093c2090d17005a76b9fe001d414c9597c0c3ed2a6cb37c4b97a2767eec8d3e: Status 404 returned error can't find the container with id 1093c2090d17005a76b9fe001d414c9597c0c3ed2a6cb37c4b97a2767eec8d3e
Oct 31 13:12:29.094621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.094592    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:29.376063 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.376037    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"]
Oct 31 13:12:29.378130 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:29.378099    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b71e93b_d7fc_4158_aef8_8e1cf1b6138c.slice/crio-f0efae68722819ebe0797d714a44bf8365da45437e89829ff855539750152179 WatchSource:0}: Error finding container f0efae68722819ebe0797d714a44bf8365da45437e89829ff855539750152179: Status 404 returned error can't find the container with id f0efae68722819ebe0797d714a44bf8365da45437e89829ff855539750152179
Oct 31 13:12:29.532218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.532177    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc" event={"ID":"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c","Type":"ContainerStarted","Data":"f0efae68722819ebe0797d714a44bf8365da45437e89829ff855539750152179"}
Oct 31 13:12:29.533968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.533938    5787 generic.go:356] "Generic (PLEG): container finished" podID="a47f6dd5-cd11-4f17-bce4-1497bb5b9769" containerID="3d62ad61489657ed6122b2cca961ab7cd996b4ef5105c9df28c1fa60cb871890" exitCode=0
Oct 31 13:12:29.534107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.534011    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j" event={"ID":"a47f6dd5-cd11-4f17-bce4-1497bb5b9769","Type":"ContainerDied","Data":"3d62ad61489657ed6122b2cca961ab7cd996b4ef5105c9df28c1fa60cb871890"}
Oct 31 13:12:29.535788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.535764    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj" event={"ID":"96c93acf-213a-4eaf-8cec-d9f895ea34aa","Type":"ContainerStarted","Data":"bbff58ee74be716cb24e047d86334f9979eab153adac8503b89b4c2fd4f61899"}
Oct 31 13:12:29.535895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.535813    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj" event={"ID":"96c93acf-213a-4eaf-8cec-d9f895ea34aa","Type":"ContainerStarted","Data":"1093c2090d17005a76b9fe001d414c9597c0c3ed2a6cb37c4b97a2767eec8d3e"}
Oct 31 13:12:29.575983 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.575923    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj" podStartSLOduration=1.575905825 podStartE2EDuration="1.575905825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:29.57395617 +0000 UTC m=+728.989898011" watchObservedRunningTime="2026-10-31 13:12:29.575905825 +0000 UTC m=+728.991847665"
Oct 31 13:12:29.681500 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.681468    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:29.794595 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.794556    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37892070-9deb-4766-aa50-232e30892617-kube-api-access-v5xm4\" (UniqueName: \"kubernetes.io/projected/37892070-9deb-4766-aa50-232e30892617-kube-api-access-v5xm4\") pod \"37892070-9deb-4766-aa50-232e30892617\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") "
Oct 31 13:12:29.795044 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.794613    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37892070-9deb-4766-aa50-232e30892617-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37892070-9deb-4766-aa50-232e30892617-maas-api-openshift-ca\") pod \"37892070-9deb-4766-aa50-232e30892617\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") "
Oct 31 13:12:29.795044 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.794689    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37892070-9deb-4766-aa50-232e30892617-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37892070-9deb-4766-aa50-232e30892617-maas-api-xks-ca\") pod \"37892070-9deb-4766-aa50-232e30892617\" (UID: \"37892070-9deb-4766-aa50-232e30892617\") "
Oct 31 13:12:29.795044 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.795000    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37892070-9deb-4766-aa50-232e30892617-maas-api-openshift-ca" pod "37892070-9deb-4766-aa50-232e30892617" (UID: "37892070-9deb-4766-aa50-232e30892617"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:29.795212 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.795186    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37892070-9deb-4766-aa50-232e30892617-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:29.796995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.796971    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37892070-9deb-4766-aa50-232e30892617-maas-api-xks-ca" pod "37892070-9deb-4766-aa50-232e30892617" (UID: "37892070-9deb-4766-aa50-232e30892617"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:29.796995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.796976    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37892070-9deb-4766-aa50-232e30892617-kube-api-access-v5xm4" pod "37892070-9deb-4766-aa50-232e30892617" (UID: "37892070-9deb-4766-aa50-232e30892617"). InnerVolumeSpecName "kube-api-access-v5xm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:29.896576 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.896546    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5xm4\" (UniqueName: \"kubernetes.io/projected/37892070-9deb-4766-aa50-232e30892617-kube-api-access-v5xm4\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:29.896576 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:29.896575    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37892070-9deb-4766-aa50-232e30892617-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:30.544854 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.544824    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"
Oct 31 13:12:30.544854 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.544843    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5" event={"ID":"37892070-9deb-4766-aa50-232e30892617","Type":"ContainerDied","Data":"ea0e8631ad72a59eeac118c322c459ec8a8d8032acb11b1a263beabb577d1827"}
Oct 31 13:12:30.545051 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.544905    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea0e8631ad72a59eeac118c322c459ec8a8d8032acb11b1a263beabb577d1827"
Oct 31 13:12:30.546639 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.546618    5787 generic.go:356] "Generic (PLEG): container finished" podID="96c93acf-213a-4eaf-8cec-d9f895ea34aa" containerID="bbff58ee74be716cb24e047d86334f9979eab153adac8503b89b4c2fd4f61899" exitCode=0
Oct 31 13:12:30.546759 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.546731    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj" event={"ID":"96c93acf-213a-4eaf-8cec-d9f895ea34aa","Type":"ContainerDied","Data":"bbff58ee74be716cb24e047d86334f9979eab153adac8503b89b4c2fd4f61899"}
Oct 31 13:12:30.548468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.548438    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc" event={"ID":"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c","Type":"ContainerStarted","Data":"67f28dda93654a1801ea7a0669f8c5eafaf70a47224510a7201cf41eb2b457f8"}
Oct 31 13:12:30.707465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.707436    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:30.804572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.804487    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-openshift-ca\") pod \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") "
Oct 31 13:12:30.804572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.804531    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-kube-api-access-dqwwp\" (UniqueName: \"kubernetes.io/projected/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-kube-api-access-dqwwp\") pod \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") "
Oct 31 13:12:30.804572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.804560    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-xks-ca\") pod \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\" (UID: \"a47f6dd5-cd11-4f17-bce4-1497bb5b9769\") "
Oct 31 13:12:30.805123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.804820    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-openshift-ca" pod "a47f6dd5-cd11-4f17-bce4-1497bb5b9769" (UID: "a47f6dd5-cd11-4f17-bce4-1497bb5b9769"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:30.805188 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.805144    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:30.806813 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.806788    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-xks-ca" pod "a47f6dd5-cd11-4f17-bce4-1497bb5b9769" (UID: "a47f6dd5-cd11-4f17-bce4-1497bb5b9769"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:30.806940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.806923    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-kube-api-access-dqwwp" pod "a47f6dd5-cd11-4f17-bce4-1497bb5b9769" (UID: "a47f6dd5-cd11-4f17-bce4-1497bb5b9769"). InnerVolumeSpecName "kube-api-access-dqwwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:30.906632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.906593    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqwwp\" (UniqueName: \"kubernetes.io/projected/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-kube-api-access-dqwwp\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:30.906632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:30.906625    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a47f6dd5-cd11-4f17-bce4-1497bb5b9769-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:31.555147 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.555101    5787 generic.go:356] "Generic (PLEG): container finished" podID="5b71e93b-d7fc-4158-aef8-8e1cf1b6138c" containerID="67f28dda93654a1801ea7a0669f8c5eafaf70a47224510a7201cf41eb2b457f8" exitCode=0
Oct 31 13:12:31.555147 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.555141    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc" event={"ID":"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c","Type":"ContainerDied","Data":"67f28dda93654a1801ea7a0669f8c5eafaf70a47224510a7201cf41eb2b457f8"}
Oct 31 13:12:31.556655 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.556633    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"
Oct 31 13:12:31.556810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.556745    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j" event={"ID":"a47f6dd5-cd11-4f17-bce4-1497bb5b9769","Type":"ContainerDied","Data":"dcfde739e8323c90500dab262fbc96bc8097ed58f36d5d1588fe373d26e239a4"}
Oct 31 13:12:31.556810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.556794    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcfde739e8323c90500dab262fbc96bc8097ed58f36d5d1588fe373d26e239a4"
Oct 31 13:12:31.759127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.759091    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:31.761236 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.761212    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:31.814319 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.814228    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-openshift-ca\") pod \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") "
Oct 31 13:12:31.815489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.815464    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-openshift-ca\") pod \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") "
Oct 31 13:12:31.815695 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.815675    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-xks-ca\") pod \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") "
Oct 31 13:12:31.815898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.815880    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-kube-api-access-q67qt\" (UniqueName: \"kubernetes.io/projected/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-kube-api-access-q67qt\") pod \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\" (UID: \"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c\") "
Oct 31 13:12:31.816054 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.816038    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96c93acf-213a-4eaf-8cec-d9f895ea34aa-kube-api-access-f9pg8\" (UniqueName: \"kubernetes.io/projected/96c93acf-213a-4eaf-8cec-d9f895ea34aa-kube-api-access-f9pg8\") pod \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") "
Oct 31 13:12:31.816172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.816157    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-xks-ca\") pod \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\" (UID: \"96c93acf-213a-4eaf-8cec-d9f895ea34aa\") "
Oct 31 13:12:31.820561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.815264    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-openshift-ca" pod "96c93acf-213a-4eaf-8cec-d9f895ea34aa" (UID: "96c93acf-213a-4eaf-8cec-d9f895ea34aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:31.820561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.820222    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-xks-ca" pod "96c93acf-213a-4eaf-8cec-d9f895ea34aa" (UID: "96c93acf-213a-4eaf-8cec-d9f895ea34aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:31.820781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.820624    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-openshift-ca" pod "5b71e93b-d7fc-4158-aef8-8e1cf1b6138c" (UID: "5b71e93b-d7fc-4158-aef8-8e1cf1b6138c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:31.824337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.824306    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96c93acf-213a-4eaf-8cec-d9f895ea34aa-kube-api-access-f9pg8" pod "96c93acf-213a-4eaf-8cec-d9f895ea34aa" (UID: "96c93acf-213a-4eaf-8cec-d9f895ea34aa"). InnerVolumeSpecName "kube-api-access-f9pg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:31.828243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.828108    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-xks-ca" pod "5b71e93b-d7fc-4158-aef8-8e1cf1b6138c" (UID: "5b71e93b-d7fc-4158-aef8-8e1cf1b6138c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:31.829621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.829591    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-kube-api-access-q67qt" pod "5b71e93b-d7fc-4158-aef8-8e1cf1b6138c" (UID: "5b71e93b-d7fc-4158-aef8-8e1cf1b6138c"). InnerVolumeSpecName "kube-api-access-q67qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:31.918397 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.918351    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q67qt\" (UniqueName: \"kubernetes.io/projected/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-kube-api-access-q67qt\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:31.918397 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.918398    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9pg8\" (UniqueName: \"kubernetes.io/projected/96c93acf-213a-4eaf-8cec-d9f895ea34aa-kube-api-access-f9pg8\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:31.918602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.918414    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:31.918602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.918429    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96c93acf-213a-4eaf-8cec-d9f895ea34aa-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:31.918602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.918443    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:31.918602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:31.918457    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:32.563430 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:32.563398    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"
Oct 31 13:12:32.563623 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:32.563435    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj" event={"ID":"96c93acf-213a-4eaf-8cec-d9f895ea34aa","Type":"ContainerDied","Data":"1093c2090d17005a76b9fe001d414c9597c0c3ed2a6cb37c4b97a2767eec8d3e"}
Oct 31 13:12:32.563623 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:32.563525    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1093c2090d17005a76b9fe001d414c9597c0c3ed2a6cb37c4b97a2767eec8d3e"
Oct 31 13:12:32.564855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:32.564826    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"
Oct 31 13:12:32.564985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:32.564858    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc" event={"ID":"5b71e93b-d7fc-4158-aef8-8e1cf1b6138c","Type":"ContainerDied","Data":"f0efae68722819ebe0797d714a44bf8365da45437e89829ff855539750152179"}
Oct 31 13:12:32.564985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:32.564909    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0efae68722819ebe0797d714a44bf8365da45437e89829ff855539750152179"
Oct 31 13:12:33.501685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.501650    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"]
Oct 31 13:12:33.502142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502111    5787 cpu_manager.go:414] "containerMap: removing container" podUID="37892070-9deb-4766-aa50-232e30892617" containerName="revoke-keys"
Oct 31 13:12:33.502142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502123    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37892070-9deb-4766-aa50-232e30892617" containerName="revoke-keys"
Oct 31 13:12:33.502214 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502159    5787 cpu_manager.go:414] "containerMap: removing container" podUID="96c93acf-213a-4eaf-8cec-d9f895ea34aa" containerName="revoke-keys"
Oct 31 13:12:33.502214 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502164    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96c93acf-213a-4eaf-8cec-d9f895ea34aa" containerName="revoke-keys"
Oct 31 13:12:33.502214 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502180    5787 cpu_manager.go:414] "containerMap: removing container" podUID="a47f6dd5-cd11-4f17-bce4-1497bb5b9769" containerName="revoke-keys"
Oct 31 13:12:33.502214 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502185    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a47f6dd5-cd11-4f17-bce4-1497bb5b9769" containerName="revoke-keys"
Oct 31 13:12:33.502214 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502191    5787 cpu_manager.go:414] "containerMap: removing container" podUID="5b71e93b-d7fc-4158-aef8-8e1cf1b6138c" containerName="revoke-keys"
Oct 31 13:12:33.502214 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502197    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b71e93b-d7fc-4158-aef8-8e1cf1b6138c" containerName="revoke-keys"
Oct 31 13:12:33.502658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502259    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="a47f6dd5-cd11-4f17-bce4-1497bb5b9769" containerName="revoke-keys"
Oct 31 13:12:33.502658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502267    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b71e93b-d7fc-4158-aef8-8e1cf1b6138c" containerName="revoke-keys"
Oct 31 13:12:33.502658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502275    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="37892070-9deb-4766-aa50-232e30892617" containerName="revoke-keys"
Oct 31 13:12:33.502658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.502285    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="96c93acf-213a-4eaf-8cec-d9f895ea34aa" containerName="revoke-keys"
Oct 31 13:12:33.505549 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.505530    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.513192 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.513158    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:33.513305 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.513224    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:12:33.513305 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.513275    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:12:33.513537 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.513492    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:12:33.516691 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.516612    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"]
Oct 31 13:12:33.637296 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.637267    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12700de6-69qm5\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.637458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.637320    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brdks\" (UniqueName: \"kubernetes.io/projected/43d6c6bf-6c92-4e1c-8271-10f521baa944-kube-api-access-brdks\") pod \"maas-api-revoke-sub-12700de6-69qm5\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.637528 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.637446    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12700de6-69qm5\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.739375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.739339    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12700de6-69qm5\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.739546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.739391    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brdks\" (UniqueName: \"kubernetes.io/projected/43d6c6bf-6c92-4e1c-8271-10f521baa944-kube-api-access-brdks\") pod \"maas-api-revoke-sub-12700de6-69qm5\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.739546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.739423    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12700de6-69qm5\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.740057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.740033    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12700de6-69qm5\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.741951 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.741926    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12700de6-69qm5\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.753541 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.753481    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brdks\" (UniqueName: \"kubernetes.io/projected/43d6c6bf-6c92-4e1c-8271-10f521baa944-kube-api-access-brdks\") pod \"maas-api-revoke-sub-12700de6-69qm5\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:33.821308 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:33.821278    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:34.107675 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.107581    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"]
Oct 31 13:12:34.112067 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:34.112038    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d6c6bf_6c92_4e1c_8271_10f521baa944.slice/crio-bc02e3c8c30850086cf1ea640727548e0b3d9f5ea4291a104b17c8b5b9b93f80 WatchSource:0}: Error finding container bc02e3c8c30850086cf1ea640727548e0b3d9f5ea4291a104b17c8b5b9b93f80: Status 404 returned error can't find the container with id bc02e3c8c30850086cf1ea640727548e0b3d9f5ea4291a104b17c8b5b9b93f80
Oct 31 13:12:34.578649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.578614    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5" event={"ID":"43d6c6bf-6c92-4e1c-8271-10f521baa944","Type":"ContainerStarted","Data":"f56d4bb473b49af12b1bf1f070287c11966cde7ae3d686e2166214802ea0ec4e"}
Oct 31 13:12:34.579069 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.578664    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5" event={"ID":"43d6c6bf-6c92-4e1c-8271-10f521baa944","Type":"ContainerStarted","Data":"bc02e3c8c30850086cf1ea640727548e0b3d9f5ea4291a104b17c8b5b9b93f80"}
Oct 31 13:12:34.602304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.602256    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5" podStartSLOduration=1.602237613 podStartE2EDuration="1.602237613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:34.601614085 +0000 UTC m=+734.017555927" watchObservedRunningTime="2026-10-31 13:12:34.602237613 +0000 UTC m=+734.018179454"
Oct 31 13:12:34.863299 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.863210    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"]
Oct 31 13:12:34.867082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.867064    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:34.878572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.878540    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"]
Oct 31 13:12:34.954281 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.954247    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbcpp\" (UniqueName: \"kubernetes.io/projected/6dd9a17d-ddbf-426d-bc71-df1139f8823e-kube-api-access-gbcpp\") pod \"maas-api-revoke-sub-7ae3e4b5-xgrpx\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:34.954449 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.954310    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae3e4b5-xgrpx\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:34.954583 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:34.954459    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae3e4b5-xgrpx\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:35.056021 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.055991    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae3e4b5-xgrpx\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:35.056139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.056127    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbcpp\" (UniqueName: \"kubernetes.io/projected/6dd9a17d-ddbf-426d-bc71-df1139f8823e-kube-api-access-gbcpp\") pod \"maas-api-revoke-sub-7ae3e4b5-xgrpx\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:35.056189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.056173    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae3e4b5-xgrpx\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:35.056728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.056691    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ae3e4b5-xgrpx\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:35.058415 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.058397    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ae3e4b5-xgrpx\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:35.070484 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.070462    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbcpp\" (UniqueName: \"kubernetes.io/projected/6dd9a17d-ddbf-426d-bc71-df1139f8823e-kube-api-access-gbcpp\") pod \"maas-api-revoke-sub-7ae3e4b5-xgrpx\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:35.182699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.182614    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:35.463813 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.463785    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"]
Oct 31 13:12:35.465149 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:35.465122    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd9a17d_ddbf_426d_bc71_df1139f8823e.slice/crio-eabe719d5955ed91152a41c8537261a57ef6b808210a13fcc55d2ce40c295f5f WatchSource:0}: Error finding container eabe719d5955ed91152a41c8537261a57ef6b808210a13fcc55d2ce40c295f5f: Status 404 returned error can't find the container with id eabe719d5955ed91152a41c8537261a57ef6b808210a13fcc55d2ce40c295f5f
Oct 31 13:12:35.585684 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.585651    5787 generic.go:356] "Generic (PLEG): container finished" podID="43d6c6bf-6c92-4e1c-8271-10f521baa944" containerID="f56d4bb473b49af12b1bf1f070287c11966cde7ae3d686e2166214802ea0ec4e" exitCode=0
Oct 31 13:12:35.586148 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.585730    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5" event={"ID":"43d6c6bf-6c92-4e1c-8271-10f521baa944","Type":"ContainerDied","Data":"f56d4bb473b49af12b1bf1f070287c11966cde7ae3d686e2166214802ea0ec4e"}
Oct 31 13:12:35.587057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:35.587035    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx" event={"ID":"6dd9a17d-ddbf-426d-bc71-df1139f8823e","Type":"ContainerStarted","Data":"eabe719d5955ed91152a41c8537261a57ef6b808210a13fcc55d2ce40c295f5f"}
Oct 31 13:12:36.648036 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:36.647917    5787 generic.go:356] "Generic (PLEG): container finished" podID="6dd9a17d-ddbf-426d-bc71-df1139f8823e" containerID="482a072fca26f4232a9b7d8d85d9a0375de835a93b1d26b0143b664bc400e675" exitCode=0
Oct 31 13:12:36.649261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:36.649220    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx" event={"ID":"6dd9a17d-ddbf-426d-bc71-df1139f8823e","Type":"ContainerDied","Data":"482a072fca26f4232a9b7d8d85d9a0375de835a93b1d26b0143b664bc400e675"}
Oct 31 13:12:36.848901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:36.848863    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:37.003296 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.003265    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-xks-ca\") pod \"43d6c6bf-6c92-4e1c-8271-10f521baa944\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") "
Oct 31 13:12:37.003468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.003317    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-openshift-ca\") pod \"43d6c6bf-6c92-4e1c-8271-10f521baa944\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") "
Oct 31 13:12:37.003468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.003374    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d6c6bf-6c92-4e1c-8271-10f521baa944-kube-api-access-brdks\" (UniqueName: \"kubernetes.io/projected/43d6c6bf-6c92-4e1c-8271-10f521baa944-kube-api-access-brdks\") pod \"43d6c6bf-6c92-4e1c-8271-10f521baa944\" (UID: \"43d6c6bf-6c92-4e1c-8271-10f521baa944\") "
Oct 31 13:12:37.003724 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.003678    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-openshift-ca" pod "43d6c6bf-6c92-4e1c-8271-10f521baa944" (UID: "43d6c6bf-6c92-4e1c-8271-10f521baa944"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:37.005664 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.005641    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-xks-ca" pod "43d6c6bf-6c92-4e1c-8271-10f521baa944" (UID: "43d6c6bf-6c92-4e1c-8271-10f521baa944"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:37.005664 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.005649    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d6c6bf-6c92-4e1c-8271-10f521baa944-kube-api-access-brdks" pod "43d6c6bf-6c92-4e1c-8271-10f521baa944" (UID: "43d6c6bf-6c92-4e1c-8271-10f521baa944"). InnerVolumeSpecName "kube-api-access-brdks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:37.104900 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.104849    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:37.104900 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.104900    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d6c6bf-6c92-4e1c-8271-10f521baa944-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:37.105107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.104918    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brdks\" (UniqueName: \"kubernetes.io/projected/43d6c6bf-6c92-4e1c-8271-10f521baa944-kube-api-access-brdks\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:37.658928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.658896    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"
Oct 31 13:12:37.658928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.658913    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5" event={"ID":"43d6c6bf-6c92-4e1c-8271-10f521baa944","Type":"ContainerDied","Data":"bc02e3c8c30850086cf1ea640727548e0b3d9f5ea4291a104b17c8b5b9b93f80"}
Oct 31 13:12:37.659463 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.658970    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc02e3c8c30850086cf1ea640727548e0b3d9f5ea4291a104b17c8b5b9b93f80"
Oct 31 13:12:37.820482 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.820453    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:37.913971 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.913877    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-xks-ca\") pod \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") "
Oct 31 13:12:37.914209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.914072    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-openshift-ca\") pod \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") "
Oct 31 13:12:37.914209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.914119    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dd9a17d-ddbf-426d-bc71-df1139f8823e-kube-api-access-gbcpp\" (UniqueName: \"kubernetes.io/projected/6dd9a17d-ddbf-426d-bc71-df1139f8823e-kube-api-access-gbcpp\") pod \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\" (UID: \"6dd9a17d-ddbf-426d-bc71-df1139f8823e\") "
Oct 31 13:12:37.914447 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.914420    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-openshift-ca" pod "6dd9a17d-ddbf-426d-bc71-df1139f8823e" (UID: "6dd9a17d-ddbf-426d-bc71-df1139f8823e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:37.914570 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.914551    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:37.916200 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.916170    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-xks-ca" pod "6dd9a17d-ddbf-426d-bc71-df1139f8823e" (UID: "6dd9a17d-ddbf-426d-bc71-df1139f8823e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:37.916300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:37.916244    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd9a17d-ddbf-426d-bc71-df1139f8823e-kube-api-access-gbcpp" pod "6dd9a17d-ddbf-426d-bc71-df1139f8823e" (UID: "6dd9a17d-ddbf-426d-bc71-df1139f8823e"). InnerVolumeSpecName "kube-api-access-gbcpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:38.015744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:38.015685    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbcpp\" (UniqueName: \"kubernetes.io/projected/6dd9a17d-ddbf-426d-bc71-df1139f8823e-kube-api-access-gbcpp\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:38.015744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:38.015745    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dd9a17d-ddbf-426d-bc71-df1139f8823e-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:38.665717 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:38.665676    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"
Oct 31 13:12:38.666120 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:38.665697    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx" event={"ID":"6dd9a17d-ddbf-426d-bc71-df1139f8823e","Type":"ContainerDied","Data":"eabe719d5955ed91152a41c8537261a57ef6b808210a13fcc55d2ce40c295f5f"}
Oct 31 13:12:38.666120 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:38.665786    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eabe719d5955ed91152a41c8537261a57ef6b808210a13fcc55d2ce40c295f5f"
Oct 31 13:12:40.786158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.786115    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"]
Oct 31 13:12:40.786656 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.786585    5787 cpu_manager.go:414] "containerMap: removing container" podUID="6dd9a17d-ddbf-426d-bc71-df1139f8823e" containerName="revoke-keys"
Oct 31 13:12:40.786656 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.786596    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dd9a17d-ddbf-426d-bc71-df1139f8823e" containerName="revoke-keys"
Oct 31 13:12:40.786656 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.786616    5787 cpu_manager.go:414] "containerMap: removing container" podUID="43d6c6bf-6c92-4e1c-8271-10f521baa944" containerName="revoke-keys"
Oct 31 13:12:40.786656 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.786622    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d6c6bf-6c92-4e1c-8271-10f521baa944" containerName="revoke-keys"
Oct 31 13:12:40.786905 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.786715    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dd9a17d-ddbf-426d-bc71-df1139f8823e" containerName="revoke-keys"
Oct 31 13:12:40.786905 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.786726    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d6c6bf-6c92-4e1c-8271-10f521baa944" containerName="revoke-keys"
Oct 31 13:12:40.790301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.790275    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:40.799752 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.799729    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:12:40.799931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.799738    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:40.799931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.799764    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:12:40.799931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.799856    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:12:40.801557 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.801534    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"]
Oct 31 13:12:40.954999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.954958    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b1ea3b-sr9dg\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:40.955155 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.955119    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsxgx\" (UniqueName: \"kubernetes.io/projected/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-kube-api-access-vsxgx\") pod \"maas-api-revoke-sub-98b1ea3b-sr9dg\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:40.955193 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:40.955171    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b1ea3b-sr9dg\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:41.056347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.056260    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsxgx\" (UniqueName: \"kubernetes.io/projected/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-kube-api-access-vsxgx\") pod \"maas-api-revoke-sub-98b1ea3b-sr9dg\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:41.056347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.056309    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b1ea3b-sr9dg\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:41.056540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.056369    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b1ea3b-sr9dg\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:41.057007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.056984    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98b1ea3b-sr9dg\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:41.058947 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.058924    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98b1ea3b-sr9dg\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:41.070795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.070764    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsxgx\" (UniqueName: \"kubernetes.io/projected/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-kube-api-access-vsxgx\") pod \"maas-api-revoke-sub-98b1ea3b-sr9dg\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:41.122375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.122337    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:41.416460 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.416355    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"]
Oct 31 13:12:41.451639 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:41.451605    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d88165_8e3b_4486_9f0b_2ac58cb80b29.slice/crio-4048a2df46f10bf67b3a0f67ba1c4578528da7b32e5cab0c06b14286d075a791 WatchSource:0}: Error finding container 4048a2df46f10bf67b3a0f67ba1c4578528da7b32e5cab0c06b14286d075a791: Status 404 returned error can't find the container with id 4048a2df46f10bf67b3a0f67ba1c4578528da7b32e5cab0c06b14286d075a791
Oct 31 13:12:41.728516 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.728473    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg" event={"ID":"c1d88165-8e3b-4486-9f0b-2ac58cb80b29","Type":"ContainerStarted","Data":"4048a2df46f10bf67b3a0f67ba1c4578528da7b32e5cab0c06b14286d075a791"}
Oct 31 13:12:41.883124 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.883091    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"]
Oct 31 13:12:41.887017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.886997    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:41.900006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.899971    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"]
Oct 31 13:12:41.967221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.967181    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549f0286-gs7bd\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:41.967221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.967219    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhd24\" (UniqueName: \"kubernetes.io/projected/a4b605c1-f2f0-4dd9-b26c-32dece01d557-kube-api-access-zhd24\") pod \"maas-api-revoke-sub-549f0286-gs7bd\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:41.967478 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:41.967348    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549f0286-gs7bd\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:42.068627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.068551    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549f0286-gs7bd\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:42.068627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.068588    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhd24\" (UniqueName: \"kubernetes.io/projected/a4b605c1-f2f0-4dd9-b26c-32dece01d557-kube-api-access-zhd24\") pod \"maas-api-revoke-sub-549f0286-gs7bd\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:42.068876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.068662    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549f0286-gs7bd\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:42.069359 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.069332    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-549f0286-gs7bd\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:42.071148 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.071127    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-xks-ca\") pod \"maas-api-revoke-sub-549f0286-gs7bd\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:42.081993 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.081971    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhd24\" (UniqueName: \"kubernetes.io/projected/a4b605c1-f2f0-4dd9-b26c-32dece01d557-kube-api-access-zhd24\") pod \"maas-api-revoke-sub-549f0286-gs7bd\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:42.200544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.200509    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:42.485826 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.485799    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"]
Oct 31 13:12:42.487262 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:42.487238    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b605c1_f2f0_4dd9_b26c_32dece01d557.slice/crio-c09f1e69dbc8e3730d9f2781d5e71a5cce3c6db9295f131d837aea02d2abfa9b WatchSource:0}: Error finding container c09f1e69dbc8e3730d9f2781d5e71a5cce3c6db9295f131d837aea02d2abfa9b: Status 404 returned error can't find the container with id c09f1e69dbc8e3730d9f2781d5e71a5cce3c6db9295f131d837aea02d2abfa9b
Oct 31 13:12:42.736772 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.736647    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd" event={"ID":"a4b605c1-f2f0-4dd9-b26c-32dece01d557","Type":"ContainerStarted","Data":"a23c92c10f0a4afd0153ce50fbd229158bc48e2e4304f61ee4460decc98b4782"}
Oct 31 13:12:42.736772 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.736726    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd" event={"ID":"a4b605c1-f2f0-4dd9-b26c-32dece01d557","Type":"ContainerStarted","Data":"c09f1e69dbc8e3730d9f2781d5e71a5cce3c6db9295f131d837aea02d2abfa9b"}
Oct 31 13:12:42.738632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.738573    5787 generic.go:356] "Generic (PLEG): container finished" podID="c1d88165-8e3b-4486-9f0b-2ac58cb80b29" containerID="73f0a9ccd50efb14539d9dd72bc9181428bf34d8741ba5cf6a5065e8b7e55f36" exitCode=0
Oct 31 13:12:42.738766 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.738630    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg" event={"ID":"c1d88165-8e3b-4486-9f0b-2ac58cb80b29","Type":"ContainerDied","Data":"73f0a9ccd50efb14539d9dd72bc9181428bf34d8741ba5cf6a5065e8b7e55f36"}
Oct 31 13:12:42.764073 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.764018    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd" podStartSLOduration=1.7640043699999999 podStartE2EDuration="1.76400437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:42.761642546 +0000 UTC m=+742.177584388" watchObservedRunningTime="2026-10-31 13:12:42.76400437 +0000 UTC m=+742.179946210"
Oct 31 13:12:42.785553 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.784754    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"]
Oct 31 13:12:42.788897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.788864    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:42.801241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.801203    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"]
Oct 31 13:12:42.878818 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.878772    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71bc3bd7-r5w56\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:42.878978 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.878842    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71bc3bd7-r5w56\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:42.879040 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.878982    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg4zn\" (UniqueName: \"kubernetes.io/projected/0645129c-18a5-43c3-9227-dc509a9677a9-kube-api-access-vg4zn\") pod \"maas-api-revoke-sub-71bc3bd7-r5w56\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:42.980661 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.980618    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71bc3bd7-r5w56\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:42.981141 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.980685    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71bc3bd7-r5w56\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:42.981141 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.980762    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg4zn\" (UniqueName: \"kubernetes.io/projected/0645129c-18a5-43c3-9227-dc509a9677a9-kube-api-access-vg4zn\") pod \"maas-api-revoke-sub-71bc3bd7-r5w56\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:42.981351 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.981329    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71bc3bd7-r5w56\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:42.983276 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.983257    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71bc3bd7-r5w56\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:42.994743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:42.994656    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg4zn\" (UniqueName: \"kubernetes.io/projected/0645129c-18a5-43c3-9227-dc509a9677a9-kube-api-access-vg4zn\") pod \"maas-api-revoke-sub-71bc3bd7-r5w56\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:43.107831 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:43.107770    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:43.463618 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:43.463583    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"]
Oct 31 13:12:43.465019 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:43.464988    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0645129c_18a5_43c3_9227_dc509a9677a9.slice/crio-70c72cd87106b091742743ec007c637b27f0f1ad2ef181231269920ef580f359 WatchSource:0}: Error finding container 70c72cd87106b091742743ec007c637b27f0f1ad2ef181231269920ef580f359: Status 404 returned error can't find the container with id 70c72cd87106b091742743ec007c637b27f0f1ad2ef181231269920ef580f359
Oct 31 13:12:43.751731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:43.751529    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56" event={"ID":"0645129c-18a5-43c3-9227-dc509a9677a9","Type":"ContainerStarted","Data":"70c72cd87106b091742743ec007c637b27f0f1ad2ef181231269920ef580f359"}
Oct 31 13:12:43.755731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:43.753976    5787 generic.go:356] "Generic (PLEG): container finished" podID="a4b605c1-f2f0-4dd9-b26c-32dece01d557" containerID="a23c92c10f0a4afd0153ce50fbd229158bc48e2e4304f61ee4460decc98b4782" exitCode=0
Oct 31 13:12:43.755731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:43.754780    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd" event={"ID":"a4b605c1-f2f0-4dd9-b26c-32dece01d557","Type":"ContainerDied","Data":"a23c92c10f0a4afd0153ce50fbd229158bc48e2e4304f61ee4460decc98b4782"}
Oct 31 13:12:43.985621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:43.985511    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:44.102440 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.102407    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-openshift-ca\") pod \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") "
Oct 31 13:12:44.102440 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.102454    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-kube-api-access-vsxgx\" (UniqueName: \"kubernetes.io/projected/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-kube-api-access-vsxgx\") pod \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") "
Oct 31 13:12:44.102674 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.102487    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-xks-ca\") pod \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\" (UID: \"c1d88165-8e3b-4486-9f0b-2ac58cb80b29\") "
Oct 31 13:12:44.102829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.102806    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-openshift-ca" pod "c1d88165-8e3b-4486-9f0b-2ac58cb80b29" (UID: "c1d88165-8e3b-4486-9f0b-2ac58cb80b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:44.102995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.102976    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:44.104992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.104962    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-kube-api-access-vsxgx" pod "c1d88165-8e3b-4486-9f0b-2ac58cb80b29" (UID: "c1d88165-8e3b-4486-9f0b-2ac58cb80b29"). InnerVolumeSpecName "kube-api-access-vsxgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:44.105242 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.105208    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-xks-ca" pod "c1d88165-8e3b-4486-9f0b-2ac58cb80b29" (UID: "c1d88165-8e3b-4486-9f0b-2ac58cb80b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:44.204143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.204105    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsxgx\" (UniqueName: \"kubernetes.io/projected/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-kube-api-access-vsxgx\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:44.204143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.204136    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1d88165-8e3b-4486-9f0b-2ac58cb80b29-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:44.762246 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.762215    5787 generic.go:356] "Generic (PLEG): container finished" podID="0645129c-18a5-43c3-9227-dc509a9677a9" containerID="825784839eaaa097a801cc7bf8d4ccdfe4dfc877654c1b7a4dcd8a31a34b994b" exitCode=0
Oct 31 13:12:44.762455 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.762278    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56" event={"ID":"0645129c-18a5-43c3-9227-dc509a9677a9","Type":"ContainerDied","Data":"825784839eaaa097a801cc7bf8d4ccdfe4dfc877654c1b7a4dcd8a31a34b994b"}
Oct 31 13:12:44.763829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.763794    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg" event={"ID":"c1d88165-8e3b-4486-9f0b-2ac58cb80b29","Type":"ContainerDied","Data":"4048a2df46f10bf67b3a0f67ba1c4578528da7b32e5cab0c06b14286d075a791"}
Oct 31 13:12:44.763958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.763845    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4048a2df46f10bf67b3a0f67ba1c4578528da7b32e5cab0c06b14286d075a791"
Oct 31 13:12:44.763958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.763947    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"
Oct 31 13:12:44.906080 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:44.906053    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:45.011921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.011820    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-xks-ca\") pod \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") "
Oct 31 13:12:45.012294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.012001    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-openshift-ca\") pod \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") "
Oct 31 13:12:45.012294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.012052    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4b605c1-f2f0-4dd9-b26c-32dece01d557-kube-api-access-zhd24\" (UniqueName: \"kubernetes.io/projected/a4b605c1-f2f0-4dd9-b26c-32dece01d557-kube-api-access-zhd24\") pod \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\" (UID: \"a4b605c1-f2f0-4dd9-b26c-32dece01d557\") "
Oct 31 13:12:45.012424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.012331    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-openshift-ca" pod "a4b605c1-f2f0-4dd9-b26c-32dece01d557" (UID: "a4b605c1-f2f0-4dd9-b26c-32dece01d557"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:45.012540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.012514    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:45.014309 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.014280    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4b605c1-f2f0-4dd9-b26c-32dece01d557-kube-api-access-zhd24" pod "a4b605c1-f2f0-4dd9-b26c-32dece01d557" (UID: "a4b605c1-f2f0-4dd9-b26c-32dece01d557"). InnerVolumeSpecName "kube-api-access-zhd24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:45.014426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.014376    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-xks-ca" pod "a4b605c1-f2f0-4dd9-b26c-32dece01d557" (UID: "a4b605c1-f2f0-4dd9-b26c-32dece01d557"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:45.110343 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.110283    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"]
Oct 31 13:12:45.110837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.110822    5787 cpu_manager.go:414] "containerMap: removing container" podUID="a4b605c1-f2f0-4dd9-b26c-32dece01d557" containerName="revoke-keys"
Oct 31 13:12:45.110881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.110839    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4b605c1-f2f0-4dd9-b26c-32dece01d557" containerName="revoke-keys"
Oct 31 13:12:45.110881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.110861    5787 cpu_manager.go:414] "containerMap: removing container" podUID="c1d88165-8e3b-4486-9f0b-2ac58cb80b29" containerName="revoke-keys"
Oct 31 13:12:45.110881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.110867    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d88165-8e3b-4486-9f0b-2ac58cb80b29" containerName="revoke-keys"
Oct 31 13:12:45.110971 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.110955    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4b605c1-f2f0-4dd9-b26c-32dece01d557" containerName="revoke-keys"
Oct 31 13:12:45.110971 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.110964    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d88165-8e3b-4486-9f0b-2ac58cb80b29" containerName="revoke-keys"
Oct 31 13:12:45.113796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.113773    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhd24\" (UniqueName: \"kubernetes.io/projected/a4b605c1-f2f0-4dd9-b26c-32dece01d557-kube-api-access-zhd24\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:45.113851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.113801    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4b605c1-f2f0-4dd9-b26c-32dece01d557-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:45.114281 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.114266    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.124914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.124875    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"]
Oct 31 13:12:45.214428 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.214384    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f31cd2ce-dbfvd\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.214587 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.214448    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd2t5\" (UniqueName: \"kubernetes.io/projected/6d670a58-5e6a-4e27-af41-921f20b393a2-kube-api-access-sd2t5\") pod \"maas-api-revoke-sub-f31cd2ce-dbfvd\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.214587 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.214507    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f31cd2ce-dbfvd\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.315593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.315504    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f31cd2ce-dbfvd\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.315593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.315564    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2t5\" (UniqueName: \"kubernetes.io/projected/6d670a58-5e6a-4e27-af41-921f20b393a2-kube-api-access-sd2t5\") pod \"maas-api-revoke-sub-f31cd2ce-dbfvd\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.315837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.315581    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f31cd2ce-dbfvd\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.316304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.316278    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f31cd2ce-dbfvd\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.318277 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.318249    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f31cd2ce-dbfvd\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.335153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.335123    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2t5\" (UniqueName: \"kubernetes.io/projected/6d670a58-5e6a-4e27-af41-921f20b393a2-kube-api-access-sd2t5\") pod \"maas-api-revoke-sub-f31cd2ce-dbfvd\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.429888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.429846    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:45.743719 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.743665    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"]
Oct 31 13:12:45.773614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.773575    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd" event={"ID":"6d670a58-5e6a-4e27-af41-921f20b393a2","Type":"ContainerStarted","Data":"c0e06b5ff93b1ddda8f325e4e49255f588e1aa2e802ea9df8ca0bef5a4087a24"}
Oct 31 13:12:45.775283 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.775258    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"
Oct 31 13:12:45.775421 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.775281    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd" event={"ID":"a4b605c1-f2f0-4dd9-b26c-32dece01d557","Type":"ContainerDied","Data":"c09f1e69dbc8e3730d9f2781d5e71a5cce3c6db9295f131d837aea02d2abfa9b"}
Oct 31 13:12:45.775421 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.775324    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09f1e69dbc8e3730d9f2781d5e71a5cce3c6db9295f131d837aea02d2abfa9b"
Oct 31 13:12:45.944643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:45.944614    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:46.048438 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.048353    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-openshift-ca\") pod \"0645129c-18a5-43c3-9227-dc509a9677a9\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") "
Oct 31 13:12:46.048438 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.048400    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0645129c-18a5-43c3-9227-dc509a9677a9-kube-api-access-vg4zn\" (UniqueName: \"kubernetes.io/projected/0645129c-18a5-43c3-9227-dc509a9677a9-kube-api-access-vg4zn\") pod \"0645129c-18a5-43c3-9227-dc509a9677a9\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") "
Oct 31 13:12:46.048873 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.048510    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-xks-ca\") pod \"0645129c-18a5-43c3-9227-dc509a9677a9\" (UID: \"0645129c-18a5-43c3-9227-dc509a9677a9\") "
Oct 31 13:12:46.048873 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.048737    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-openshift-ca" pod "0645129c-18a5-43c3-9227-dc509a9677a9" (UID: "0645129c-18a5-43c3-9227-dc509a9677a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:46.048961 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.048936    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:46.050734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.050688    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-xks-ca" pod "0645129c-18a5-43c3-9227-dc509a9677a9" (UID: "0645129c-18a5-43c3-9227-dc509a9677a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:46.050798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.050740    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0645129c-18a5-43c3-9227-dc509a9677a9-kube-api-access-vg4zn" pod "0645129c-18a5-43c3-9227-dc509a9677a9" (UID: "0645129c-18a5-43c3-9227-dc509a9677a9"). InnerVolumeSpecName "kube-api-access-vg4zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:46.149815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.149788    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0645129c-18a5-43c3-9227-dc509a9677a9-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:46.149815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.149817    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg4zn\" (UniqueName: \"kubernetes.io/projected/0645129c-18a5-43c3-9227-dc509a9677a9-kube-api-access-vg4zn\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:46.731278 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.731243    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"]
Oct 31 13:12:46.731786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.731773    5787 cpu_manager.go:414] "containerMap: removing container" podUID="0645129c-18a5-43c3-9227-dc509a9677a9" containerName="revoke-keys"
Oct 31 13:12:46.731833 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.731788    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0645129c-18a5-43c3-9227-dc509a9677a9" containerName="revoke-keys"
Oct 31 13:12:46.731871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.731859    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="0645129c-18a5-43c3-9227-dc509a9677a9" containerName="revoke-keys"
Oct 31 13:12:46.735636 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.735612    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:46.744657 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.744579    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"]
Oct 31 13:12:46.756300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.756264    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c95abd5-4rkzx\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:46.756488 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.756319    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c95abd5-4rkzx\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:46.756595 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.756573    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x7cq\" (UniqueName: \"kubernetes.io/projected/18475b18-2d0a-4946-926a-812bdd8ddce0-kube-api-access-9x7cq\") pod \"maas-api-revoke-sub-0c95abd5-4rkzx\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:46.784294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.784261    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"
Oct 31 13:12:46.784489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.784257    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56" event={"ID":"0645129c-18a5-43c3-9227-dc509a9677a9","Type":"ContainerDied","Data":"70c72cd87106b091742743ec007c637b27f0f1ad2ef181231269920ef580f359"}
Oct 31 13:12:46.784489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.784385    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70c72cd87106b091742743ec007c637b27f0f1ad2ef181231269920ef580f359"
Oct 31 13:12:46.786247 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.786216    5787 generic.go:356] "Generic (PLEG): container finished" podID="6d670a58-5e6a-4e27-af41-921f20b393a2" containerID="2569dd32ca6cffd5d2266eda18654caad9e6945dc296ed65aa00fa5a4b262b48" exitCode=0
Oct 31 13:12:46.786391 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.786313    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd" event={"ID":"6d670a58-5e6a-4e27-af41-921f20b393a2","Type":"ContainerDied","Data":"2569dd32ca6cffd5d2266eda18654caad9e6945dc296ed65aa00fa5a4b262b48"}
Oct 31 13:12:46.857610 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.857572    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x7cq\" (UniqueName: \"kubernetes.io/projected/18475b18-2d0a-4946-926a-812bdd8ddce0-kube-api-access-9x7cq\") pod \"maas-api-revoke-sub-0c95abd5-4rkzx\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:46.857814 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.857731    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c95abd5-4rkzx\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:46.857814 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.857758    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c95abd5-4rkzx\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:46.858428 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.858392    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c95abd5-4rkzx\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:46.860329 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.860305    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c95abd5-4rkzx\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:46.871737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:46.871688    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x7cq\" (UniqueName: \"kubernetes.io/projected/18475b18-2d0a-4946-926a-812bdd8ddce0-kube-api-access-9x7cq\") pod \"maas-api-revoke-sub-0c95abd5-4rkzx\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:47.051511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.051474    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:47.338034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.338002    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"]
Oct 31 13:12:47.339049 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:47.339022    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18475b18_2d0a_4946_926a_812bdd8ddce0.slice/crio-39f88b87b0d847ab2ec12d4d54aaa906d08e33b42e33431239bd1980664b3646 WatchSource:0}: Error finding container 39f88b87b0d847ab2ec12d4d54aaa906d08e33b42e33431239bd1980664b3646: Status 404 returned error can't find the container with id 39f88b87b0d847ab2ec12d4d54aaa906d08e33b42e33431239bd1980664b3646
Oct 31 13:12:47.795487 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.795444    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx" event={"ID":"18475b18-2d0a-4946-926a-812bdd8ddce0","Type":"ContainerStarted","Data":"1cca89e296c01365980ef2e92a0ff1db52fbe801e61e7a83093aa1439607d252"}
Oct 31 13:12:47.795680 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.795508    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx" event={"ID":"18475b18-2d0a-4946-926a-812bdd8ddce0","Type":"ContainerStarted","Data":"39f88b87b0d847ab2ec12d4d54aaa906d08e33b42e33431239bd1980664b3646"}
Oct 31 13:12:47.816344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.816282    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx" podStartSLOduration=1.816262623 podStartE2EDuration="1.816262623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:47.815023025 +0000 UTC m=+747.230964872" watchObservedRunningTime="2026-10-31 13:12:47.816262623 +0000 UTC m=+747.232204466"
Oct 31 13:12:47.938888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.938859    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:47.971746 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.970599    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-openshift-ca\") pod \"6d670a58-5e6a-4e27-af41-921f20b393a2\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") "
Oct 31 13:12:47.971746 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.970793    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d670a58-5e6a-4e27-af41-921f20b393a2-kube-api-access-sd2t5\" (UniqueName: \"kubernetes.io/projected/6d670a58-5e6a-4e27-af41-921f20b393a2-kube-api-access-sd2t5\") pod \"6d670a58-5e6a-4e27-af41-921f20b393a2\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") "
Oct 31 13:12:47.971746 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.970889    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-xks-ca\") pod \"6d670a58-5e6a-4e27-af41-921f20b393a2\" (UID: \"6d670a58-5e6a-4e27-af41-921f20b393a2\") "
Oct 31 13:12:47.972066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.972024    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-openshift-ca" pod "6d670a58-5e6a-4e27-af41-921f20b393a2" (UID: "6d670a58-5e6a-4e27-af41-921f20b393a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:47.974161 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.974130    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d670a58-5e6a-4e27-af41-921f20b393a2-kube-api-access-sd2t5" pod "6d670a58-5e6a-4e27-af41-921f20b393a2" (UID: "6d670a58-5e6a-4e27-af41-921f20b393a2"). InnerVolumeSpecName "kube-api-access-sd2t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:47.974286 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:47.974224    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-xks-ca" pod "6d670a58-5e6a-4e27-af41-921f20b393a2" (UID: "6d670a58-5e6a-4e27-af41-921f20b393a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:48.072510 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:48.072482    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:48.072510 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:48.072509    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd2t5\" (UniqueName: \"kubernetes.io/projected/6d670a58-5e6a-4e27-af41-921f20b393a2-kube-api-access-sd2t5\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:48.073002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:48.072521    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d670a58-5e6a-4e27-af41-921f20b393a2-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:48.254318 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:12:48.254281    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:12:48.802585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:48.802551    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"
Oct 31 13:12:48.802776 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:48.802551    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd" event={"ID":"6d670a58-5e6a-4e27-af41-921f20b393a2","Type":"ContainerDied","Data":"c0e06b5ff93b1ddda8f325e4e49255f588e1aa2e802ea9df8ca0bef5a4087a24"}
Oct 31 13:12:48.802776 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:48.802678    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0e06b5ff93b1ddda8f325e4e49255f588e1aa2e802ea9df8ca0bef5a4087a24"
Oct 31 13:12:48.804227 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:48.804197    5787 generic.go:356] "Generic (PLEG): container finished" podID="18475b18-2d0a-4946-926a-812bdd8ddce0" containerID="1cca89e296c01365980ef2e92a0ff1db52fbe801e61e7a83093aa1439607d252" exitCode=0
Oct 31 13:12:48.804359 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:48.804287    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx" event={"ID":"18475b18-2d0a-4946-926a-812bdd8ddce0","Type":"ContainerDied","Data":"1cca89e296c01365980ef2e92a0ff1db52fbe801e61e7a83093aa1439607d252"}
Oct 31 13:12:49.131680 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.131556    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"]
Oct 31 13:12:49.132998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.132965    5787 cpu_manager.go:414] "containerMap: removing container" podUID="6d670a58-5e6a-4e27-af41-921f20b393a2" containerName="revoke-keys"
Oct 31 13:12:49.132998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.132997    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d670a58-5e6a-4e27-af41-921f20b393a2" containerName="revoke-keys"
Oct 31 13:12:49.133225 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.133159    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d670a58-5e6a-4e27-af41-921f20b393a2" containerName="revoke-keys"
Oct 31 13:12:49.141058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.141028    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.147536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.147506    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"]
Oct 31 13:12:49.193325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.193294    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3192b15-m6vgc\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.193476 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.193331    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxdz\" (UniqueName: \"kubernetes.io/projected/6f4efea5-a9c8-4060-a56c-717040239eac-kube-api-access-zxxdz\") pod \"maas-api-revoke-sub-b3192b15-m6vgc\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.193476 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.193353    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3192b15-m6vgc\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.294278 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.294241    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3192b15-m6vgc\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.294278 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.294282    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxdz\" (UniqueName: \"kubernetes.io/projected/6f4efea5-a9c8-4060-a56c-717040239eac-kube-api-access-zxxdz\") pod \"maas-api-revoke-sub-b3192b15-m6vgc\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.294513 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.294305    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3192b15-m6vgc\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.295107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.295077    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3192b15-m6vgc\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.296878 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.296854    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3192b15-m6vgc\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.308127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.308100    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxdz\" (UniqueName: \"kubernetes.io/projected/6f4efea5-a9c8-4060-a56c-717040239eac-kube-api-access-zxxdz\") pod \"maas-api-revoke-sub-b3192b15-m6vgc\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.458276 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.458234    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:49.741126 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.741092    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"]
Oct 31 13:12:49.742292 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:49.742261    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4efea5_a9c8_4060_a56c_717040239eac.slice/crio-7c95ea56a651369276a41f1ad2cc252bd6f0cd7f07fc0b5c1ea7c73895ccc732 WatchSource:0}: Error finding container 7c95ea56a651369276a41f1ad2cc252bd6f0cd7f07fc0b5c1ea7c73895ccc732: Status 404 returned error can't find the container with id 7c95ea56a651369276a41f1ad2cc252bd6f0cd7f07fc0b5c1ea7c73895ccc732
Oct 31 13:12:49.811349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.811318    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc" event={"ID":"6f4efea5-a9c8-4060-a56c-717040239eac","Type":"ContainerStarted","Data":"7c95ea56a651369276a41f1ad2cc252bd6f0cd7f07fc0b5c1ea7c73895ccc732"}
Oct 31 13:12:49.950360 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:49.950329    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:50.003094 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.003007    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-xks-ca\") pod \"18475b18-2d0a-4946-926a-812bdd8ddce0\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") "
Oct 31 13:12:50.003245 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.003100    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18475b18-2d0a-4946-926a-812bdd8ddce0-kube-api-access-9x7cq\" (UniqueName: \"kubernetes.io/projected/18475b18-2d0a-4946-926a-812bdd8ddce0-kube-api-access-9x7cq\") pod \"18475b18-2d0a-4946-926a-812bdd8ddce0\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") "
Oct 31 13:12:50.003293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.003274    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-openshift-ca\") pod \"18475b18-2d0a-4946-926a-812bdd8ddce0\" (UID: \"18475b18-2d0a-4946-926a-812bdd8ddce0\") "
Oct 31 13:12:50.003733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.003685    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-openshift-ca" pod "18475b18-2d0a-4946-926a-812bdd8ddce0" (UID: "18475b18-2d0a-4946-926a-812bdd8ddce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:50.003873 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.003829    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:50.006197 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.006135    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18475b18-2d0a-4946-926a-812bdd8ddce0-kube-api-access-9x7cq" pod "18475b18-2d0a-4946-926a-812bdd8ddce0" (UID: "18475b18-2d0a-4946-926a-812bdd8ddce0"). InnerVolumeSpecName "kube-api-access-9x7cq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:50.028751 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.006136    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-xks-ca" pod "18475b18-2d0a-4946-926a-812bdd8ddce0" (UID: "18475b18-2d0a-4946-926a-812bdd8ddce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:50.105240 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.105207    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18475b18-2d0a-4946-926a-812bdd8ddce0-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:50.105240 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.105237    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9x7cq\" (UniqueName: \"kubernetes.io/projected/18475b18-2d0a-4946-926a-812bdd8ddce0-kube-api-access-9x7cq\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:50.837763 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.837724    5787 generic.go:356] "Generic (PLEG): container finished" podID="6f4efea5-a9c8-4060-a56c-717040239eac" containerID="9e69b4ca314adf3766ed731c51dd80ba57f4b3e658db601d3cb7ad81c934455c" exitCode=0
Oct 31 13:12:50.838369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.838342    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc" event={"ID":"6f4efea5-a9c8-4060-a56c-717040239eac","Type":"ContainerDied","Data":"9e69b4ca314adf3766ed731c51dd80ba57f4b3e658db601d3cb7ad81c934455c"}
Oct 31 13:12:50.849155 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.849122    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"
Oct 31 13:12:50.849300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.849130    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx" event={"ID":"18475b18-2d0a-4946-926a-812bdd8ddce0","Type":"ContainerDied","Data":"39f88b87b0d847ab2ec12d4d54aaa906d08e33b42e33431239bd1980664b3646"}
Oct 31 13:12:50.849300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:50.849244    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39f88b87b0d847ab2ec12d4d54aaa906d08e33b42e33431239bd1980664b3646"
Oct 31 13:12:53.068483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.068451    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:53.120726 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.119479    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f4efea5-a9c8-4060-a56c-717040239eac-kube-api-access-zxxdz\" (UniqueName: \"kubernetes.io/projected/6f4efea5-a9c8-4060-a56c-717040239eac-kube-api-access-zxxdz\") pod \"6f4efea5-a9c8-4060-a56c-717040239eac\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") "
Oct 31 13:12:53.148745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.138928    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f4efea5-a9c8-4060-a56c-717040239eac-kube-api-access-zxxdz" pod "6f4efea5-a9c8-4060-a56c-717040239eac" (UID: "6f4efea5-a9c8-4060-a56c-717040239eac"). InnerVolumeSpecName "kube-api-access-zxxdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:12:53.230323 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.230290    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-openshift-ca\") pod \"6f4efea5-a9c8-4060-a56c-717040239eac\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") "
Oct 31 13:12:53.230660 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.230641    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-xks-ca\") pod \"6f4efea5-a9c8-4060-a56c-717040239eac\" (UID: \"6f4efea5-a9c8-4060-a56c-717040239eac\") "
Oct 31 13:12:53.231254 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.231239    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxxdz\" (UniqueName: \"kubernetes.io/projected/6f4efea5-a9c8-4060-a56c-717040239eac-kube-api-access-zxxdz\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:53.255986 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.255952    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-openshift-ca" pod "6f4efea5-a9c8-4060-a56c-717040239eac" (UID: "6f4efea5-a9c8-4060-a56c-717040239eac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:12:53.281891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.281849    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-xks-ca" pod "6f4efea5-a9c8-4060-a56c-717040239eac" (UID: "6f4efea5-a9c8-4060-a56c-717040239eac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:12:53.343740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.342526    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:53.343740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.342566    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f4efea5-a9c8-4060-a56c-717040239eac-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:12:53.994609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.994568    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"]
Oct 31 13:12:53.995990 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.995962    5787 cpu_manager.go:414] "containerMap: removing container" podUID="6f4efea5-a9c8-4060-a56c-717040239eac" containerName="revoke-keys"
Oct 31 13:12:53.995990 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.995990    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f4efea5-a9c8-4060-a56c-717040239eac" containerName="revoke-keys"
Oct 31 13:12:53.996258 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.996019    5787 cpu_manager.go:414] "containerMap: removing container" podUID="18475b18-2d0a-4946-926a-812bdd8ddce0" containerName="revoke-keys"
Oct 31 13:12:53.996258 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.996027    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18475b18-2d0a-4946-926a-812bdd8ddce0" containerName="revoke-keys"
Oct 31 13:12:53.996382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.996275    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f4efea5-a9c8-4060-a56c-717040239eac" containerName="revoke-keys"
Oct 31 13:12:53.996382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:53.996288    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="18475b18-2d0a-4946-926a-812bdd8ddce0" containerName="revoke-keys"
Oct 31 13:12:54.001065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.001043    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.023849 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.023819    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"
Oct 31 13:12:54.024025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.023827    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc" event={"ID":"6f4efea5-a9c8-4060-a56c-717040239eac","Type":"ContainerDied","Data":"7c95ea56a651369276a41f1ad2cc252bd6f0cd7f07fc0b5c1ea7c73895ccc732"}
Oct 31 13:12:54.024025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.023950    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c95ea56a651369276a41f1ad2cc252bd6f0cd7f07fc0b5c1ea7c73895ccc732"
Oct 31 13:12:54.030928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.030899    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-f9e24f\"/\"distinct-w5-f9e24f-kserve-self-signed-certs\""
Oct 31 13:12:54.049816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.049778    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"]
Oct 31 13:12:54.159796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.157546    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-dshm\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.159796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.157631    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-model-cache\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.159796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.157766    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tmp-dir\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.159796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.157802    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh52l\" (UniqueName: \"kubernetes.io/projected/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-kube-api-access-sh52l\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.159796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.157852    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-home\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.159796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.157898    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tls-certs\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.258674 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.258637    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-home\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.258901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.258685    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tls-certs\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.258901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.258739    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-dshm\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.258901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.258779    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-model-cache\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.258901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.258850    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tmp-dir\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.258901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.258884    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh52l\" (UniqueName: \"kubernetes.io/projected/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-kube-api-access-sh52l\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.259173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.259094    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-home\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.259215 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.259192    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-model-cache\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.259584 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.259561    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tmp-dir\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.261269 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.261247    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-dshm\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.261814 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.261796    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tls-certs\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.274475 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.274446    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh52l\" (UniqueName: \"kubernetes.io/projected/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-kube-api-access-sh52l\") pod \"distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") " pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.316223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.316190    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:54.611011 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:54.610976    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"]
Oct 31 13:12:54.614863 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:54.614836    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04a8e1d5_8801_435a_a98e_e4ae80cf10a6.slice/crio-c8fdb596a0752aed936c4426d22a7c6309f6d545d370f2920113af590c5d5138 WatchSource:0}: Error finding container c8fdb596a0752aed936c4426d22a7c6309f6d545d370f2920113af590c5d5138: Status 404 returned error can't find the container with id c8fdb596a0752aed936c4426d22a7c6309f6d545d370f2920113af590c5d5138
Oct 31 13:12:55.032087 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.032043    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" event={"ID":"04a8e1d5-8801-435a-a98e-e4ae80cf10a6","Type":"ContainerStarted","Data":"33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962"}
Oct 31 13:12:55.032253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.032096    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" event={"ID":"04a8e1d5-8801-435a-a98e-e4ae80cf10a6","Type":"ContainerStarted","Data":"c8fdb596a0752aed936c4426d22a7c6309f6d545d370f2920113af590c5d5138"}
Oct 31 13:12:55.057259 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.057205    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podStartSLOduration=2.057190503 podStartE2EDuration="2.057190503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:55.055693393 +0000 UTC m=+754.471635235" watchObservedRunningTime="2026-10-31 13:12:55.057190503 +0000 UTC m=+754.473132344"
Oct 31 13:12:55.317320 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.317229    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:12:55.548656 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.548621    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"]
Oct 31 13:12:55.554827 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.554800    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.568787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.568666    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"]
Oct 31 13:12:55.577797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.577765    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"]
Oct 31 13:12:55.581645 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.581616    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.604605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.604568    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"]
Oct 31 13:12:55.676729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.676660    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b850a6e-358a-4baa-8730-9b569d126952-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.676729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.676731    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d56747-87ac-4de5-a195-7d50afcee740-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.676986 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.676797    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.676986 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.676824    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkgkp\" (UniqueName: \"kubernetes.io/projected/2b850a6e-358a-4baa-8730-9b569d126952-kube-api-access-vkgkp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.676986 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.676918    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d56747-87ac-4de5-a195-7d50afcee740-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.677138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677005    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b850a6e-358a-4baa-8730-9b569d126952-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.677138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677032    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.677138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677069    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b850a6e-358a-4baa-8730-9b569d126952-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.677280 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677238    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.677329 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677278    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.677375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677352    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.677415 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677375    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88zhx\" (UniqueName: \"kubernetes.io/projected/80d56747-87ac-4de5-a195-7d50afcee740-kube-api-access-88zhx\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.677415 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677405    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.677515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677421    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.677515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677462    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d56747-87ac-4de5-a195-7d50afcee740-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.677515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677490    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.677635 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677535    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.677635 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.677563    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.779243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779206    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b850a6e-358a-4baa-8730-9b569d126952-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.779243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779247    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d56747-87ac-4de5-a195-7d50afcee740-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.779462 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779268    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.779462 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779385    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkgkp\" (UniqueName: \"kubernetes.io/projected/2b850a6e-358a-4baa-8730-9b569d126952-kube-api-access-vkgkp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.779462 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779428    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d56747-87ac-4de5-a195-7d50afcee740-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.779596 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779488    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b850a6e-358a-4baa-8730-9b569d126952-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.779596 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779525    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.779596 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779586    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b850a6e-358a-4baa-8730-9b569d126952-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.779784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779665    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.779784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779762    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.779922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779797    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.779922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779843    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.779922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779868    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88zhx\" (UniqueName: \"kubernetes.io/projected/80d56747-87ac-4de5-a195-7d50afcee740-kube-api-access-88zhx\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.779922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779918    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.780127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779947    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.780127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779961    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.780127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.779973    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d56747-87ac-4de5-a195-7d50afcee740-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.780127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780087    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b850a6e-358a-4baa-8730-9b569d126952-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.780349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780144    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-workload-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.780349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780167    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-workload-certs\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.780349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780202    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.780349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780208    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.780349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780229    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.780349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780300    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.780349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780350    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.780688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780594    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-istio-data\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.780688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780624    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80d56747-87ac-4de5-a195-7d50afcee740-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.780688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.780669    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-credential-socket\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.782787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.782761    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b850a6e-358a-4baa-8730-9b569d126952-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.782878 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.782801    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80d56747-87ac-4de5-a195-7d50afcee740-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.782878 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.782804    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80d56747-87ac-4de5-a195-7d50afcee740-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.783013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.782996    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b850a6e-358a-4baa-8730-9b569d126952-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.793125 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.793101    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80d56747-87ac-4de5-a195-7d50afcee740-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.793347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.793327    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkgkp\" (UniqueName: \"kubernetes.io/projected/2b850a6e-358a-4baa-8730-9b569d126952-kube-api-access-vkgkp\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.795595 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.795576    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88zhx\" (UniqueName: \"kubernetes.io/projected/80d56747-87ac-4de5-a195-7d50afcee740-kube-api-access-88zhx\") pod \"maas-default-gateway-openshift-default-7f5667f86c-8h4dd\" (UID: \"80d56747-87ac-4de5-a195-7d50afcee740\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.799045 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.799023    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b850a6e-358a-4baa-8730-9b569d126952-istio-token\") pod \"maas-default-gateway-openshift-default-7f5667f86c-72d7r\" (UID: \"2b850a6e-358a-4baa-8730-9b569d126952\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:55.870593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.870496    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:55.904927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:55.904878    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:56.214067 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:56.214032    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"]
Oct 31 13:12:56.214854 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:56.214827    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d56747_87ac_4de5_a195_7d50afcee740.slice/crio-54ab99fbf0dc62003fb6183dd891fe6a31bbd201ea70c2704ac2e70fbf522e06 WatchSource:0}: Error finding container 54ab99fbf0dc62003fb6183dd891fe6a31bbd201ea70c2704ac2e70fbf522e06: Status 404 returned error can't find the container with id 54ab99fbf0dc62003fb6183dd891fe6a31bbd201ea70c2704ac2e70fbf522e06
Oct 31 13:12:56.217523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:56.217489    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:12:56.217631 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:56.217568    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:12:56.217631 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:56.217614    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:12:56.245418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:56.245389    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"]
Oct 31 13:12:56.253009 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:56.252976    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b850a6e_358a_4baa_8730_9b569d126952.slice/crio-1bd196cb786eeba47c9f92d3d1091cb1e6002357b120518dfdb9512188d30a3f WatchSource:0}: Error finding container 1bd196cb786eeba47c9f92d3d1091cb1e6002357b120518dfdb9512188d30a3f: Status 404 returned error can't find the container with id 1bd196cb786eeba47c9f92d3d1091cb1e6002357b120518dfdb9512188d30a3f
Oct 31 13:12:56.256180 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:56.256146    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:12:56.256264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:56.256220    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:12:56.256264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:56.256250    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:12:57.092236 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.092194    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"]
Oct 31 13:12:57.099732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.099574    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.105511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.104118    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" event={"ID":"80d56747-87ac-4de5-a195-7d50afcee740","Type":"ContainerStarted","Data":"dc6cf61278c90490d83318f8656dceb5105fa21076e25ac4c2f72e9d3d3be4de"}
Oct 31 13:12:57.105511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.104187    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" event={"ID":"80d56747-87ac-4de5-a195-7d50afcee740","Type":"ContainerStarted","Data":"54ab99fbf0dc62003fb6183dd891fe6a31bbd201ea70c2704ac2e70fbf522e06"}
Oct 31 13:12:57.105511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.104636    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"]
Oct 31 13:12:57.106285 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.106262    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:12:57.106507 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.106374    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:12:57.106507 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.106426    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:12:57.106507 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.106493    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:12:57.107988 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.107963    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" event={"ID":"2b850a6e-358a-4baa-8730-9b569d126952","Type":"ContainerStarted","Data":"f16e13aedc9d75fe27c73517ef13835749545ee9085ebfb7b07b05d26de5909c"}
Oct 31 13:12:57.108089 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.108007    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" event={"ID":"2b850a6e-358a-4baa-8730-9b569d126952","Type":"ContainerStarted","Data":"1bd196cb786eeba47c9f92d3d1091cb1e6002357b120518dfdb9512188d30a3f"}
Oct 31 13:12:57.199918 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.199883    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69846f89-47nlm\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.200075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.200053    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69846f89-47nlm\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.200118 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.200099    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp882\" (UniqueName: \"kubernetes.io/projected/525ea675-86c9-4900-b9c1-67bc9a203896-kube-api-access-lp882\") pod \"maas-api-revoke-sub-69846f89-47nlm\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.220824 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.220753    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" podStartSLOduration=2.220733517 podStartE2EDuration="2.220733517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:57.188153185 +0000 UTC m=+756.604095059" watchObservedRunningTime="2026-10-31 13:12:57.220733517 +0000 UTC m=+756.636675358"
Oct 31 13:12:57.222289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.222242    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podStartSLOduration=2.222229965 podStartE2EDuration="2.222229965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:12:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:12:57.216208321 +0000 UTC m=+756.632150160" watchObservedRunningTime="2026-10-31 13:12:57.222229965 +0000 UTC m=+756.638171805"
Oct 31 13:12:57.301552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.301513    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69846f89-47nlm\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.301743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.301622    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69846f89-47nlm\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.301743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.301653    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp882\" (UniqueName: \"kubernetes.io/projected/525ea675-86c9-4900-b9c1-67bc9a203896-kube-api-access-lp882\") pod \"maas-api-revoke-sub-69846f89-47nlm\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.302406 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.302379    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69846f89-47nlm\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.304794 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.304364    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69846f89-47nlm\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.315569 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.315537    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp882\" (UniqueName: \"kubernetes.io/projected/525ea675-86c9-4900-b9c1-67bc9a203896-kube-api-access-lp882\") pod \"maas-api-revoke-sub-69846f89-47nlm\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.419688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.419602    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:12:57.792204 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.792167    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"]
Oct 31 13:12:57.793663 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:12:57.793614    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525ea675_86c9_4900_b9c1_67bc9a203896.slice/crio-c79abb3bbcc03546fe72973d94eab0c34ea11fc53daab87c6bae28be3e8a0c55 WatchSource:0}: Error finding container c79abb3bbcc03546fe72973d94eab0c34ea11fc53daab87c6bae28be3e8a0c55: Status 404 returned error can't find the container with id c79abb3bbcc03546fe72973d94eab0c34ea11fc53daab87c6bae28be3e8a0c55
Oct 31 13:12:57.877754 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.874125    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:12:57.880425 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.879989    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8h4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:12:57.880425 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.880057    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" podUID="80d56747-87ac-4de5-a195-7d50afcee740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:12:57.905243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.905114    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:12:57.906477 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.906366    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-72d7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 13:12:57.906477 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:57.906423    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podUID="2b850a6e-358a-4baa-8730-9b569d126952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 13:12:58.114654 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:58.114617    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm" event={"ID":"525ea675-86c9-4900-b9c1-67bc9a203896","Type":"ContainerStarted","Data":"c79abb3bbcc03546fe72973d94eab0c34ea11fc53daab87c6bae28be3e8a0c55"}
Oct 31 13:12:58.873427 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:58.872958    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8h4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:12:58.873427 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:58.873023    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" podUID="80d56747-87ac-4de5-a195-7d50afcee740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:12:58.905820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:58.905782    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-72d7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 13:12:58.905975 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:58.905846    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podUID="2b850a6e-358a-4baa-8730-9b569d126952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 13:12:59.124483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:59.124393    5787 generic.go:356] "Generic (PLEG): container finished" podID="525ea675-86c9-4900-b9c1-67bc9a203896" containerID="e1e5e6ef1de3a63d7955946e385914cd1f20f8d8b4ffd3f5bb520e3557fd0184" exitCode=0
Oct 31 13:12:59.124884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:59.124490    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm" event={"ID":"525ea675-86c9-4900-b9c1-67bc9a203896","Type":"ContainerDied","Data":"e1e5e6ef1de3a63d7955946e385914cd1f20f8d8b4ffd3f5bb520e3557fd0184"}
Oct 31 13:12:59.874731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:59.874103    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8h4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:12:59.874731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:59.874170    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" podUID="80d56747-87ac-4de5-a195-7d50afcee740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:12:59.905603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:59.905564    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-72d7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 13:12:59.905833 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:12:59.905615    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podUID="2b850a6e-358a-4baa-8730-9b569d126952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 13:13:00.063733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.063565    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"]
Oct 31 13:13:00.083008 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.082975    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.094788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.092991    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frw8h\" (UniqueName: \"kubernetes.io/projected/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-kube-api-access-frw8h\") pod \"maas-api-revoke-sub-998e19c8-5gh7s\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.094788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.093173    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-998e19c8-5gh7s\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.094788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.093445    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-998e19c8-5gh7s\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.094788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.094626    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"]
Oct 31 13:13:00.195580 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.195537    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frw8h\" (UniqueName: \"kubernetes.io/projected/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-kube-api-access-frw8h\") pod \"maas-api-revoke-sub-998e19c8-5gh7s\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.196088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.195748    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-998e19c8-5gh7s\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.196088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.195822    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-998e19c8-5gh7s\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.196806 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.196781    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-998e19c8-5gh7s\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.202041 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.202013    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-998e19c8-5gh7s\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.212339 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.212314    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frw8h\" (UniqueName: \"kubernetes.io/projected/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-kube-api-access-frw8h\") pod \"maas-api-revoke-sub-998e19c8-5gh7s\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.410200 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.409947    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:00.497535 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.497500    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:13:00.609411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.607952    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/525ea675-86c9-4900-b9c1-67bc9a203896-kube-api-access-lp882\" (UniqueName: \"kubernetes.io/projected/525ea675-86c9-4900-b9c1-67bc9a203896-kube-api-access-lp882\") pod \"525ea675-86c9-4900-b9c1-67bc9a203896\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") "
Oct 31 13:13:00.609411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.608074    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-xks-ca\") pod \"525ea675-86c9-4900-b9c1-67bc9a203896\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") "
Oct 31 13:13:00.609411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.608249    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-openshift-ca\") pod \"525ea675-86c9-4900-b9c1-67bc9a203896\" (UID: \"525ea675-86c9-4900-b9c1-67bc9a203896\") "
Oct 31 13:13:00.610340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.609973    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-openshift-ca" pod "525ea675-86c9-4900-b9c1-67bc9a203896" (UID: "525ea675-86c9-4900-b9c1-67bc9a203896"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:13:00.621733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.618033    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-xks-ca" pod "525ea675-86c9-4900-b9c1-67bc9a203896" (UID: "525ea675-86c9-4900-b9c1-67bc9a203896"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:13:00.621733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.619959    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/525ea675-86c9-4900-b9c1-67bc9a203896-kube-api-access-lp882" pod "525ea675-86c9-4900-b9c1-67bc9a203896" (UID: "525ea675-86c9-4900-b9c1-67bc9a203896"). InnerVolumeSpecName "kube-api-access-lp882". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:13:00.714031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.714003    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:00.714031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.714027    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/525ea675-86c9-4900-b9c1-67bc9a203896-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:00.714031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.714037    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp882\" (UniqueName: \"kubernetes.io/projected/525ea675-86c9-4900-b9c1-67bc9a203896-kube-api-access-lp882\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:00.886772 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.875012    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8h4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:13:00.886772 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.875097    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" podUID="80d56747-87ac-4de5-a195-7d50afcee740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:13:00.889406 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.889381    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"]
Oct 31 13:13:00.891674 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:00.891649    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7965ee70_9bad_459e_8709_ac3a1d3cbf8f.slice/crio-081c986069746faa2c66f23755cff08b8e27c9fd8fbf5ec688f4b00c66671d04 WatchSource:0}: Error finding container 081c986069746faa2c66f23755cff08b8e27c9fd8fbf5ec688f4b00c66671d04: Status 404 returned error can't find the container with id 081c986069746faa2c66f23755cff08b8e27c9fd8fbf5ec688f4b00c66671d04
Oct 31 13:13:00.905776 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.905753    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-72d7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 13:13:00.905835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:00.905806    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podUID="2b850a6e-358a-4baa-8730-9b569d126952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 13:13:01.163732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:01.154566    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s" event={"ID":"7965ee70-9bad-459e-8709-ac3a1d3cbf8f","Type":"ContainerStarted","Data":"081c986069746faa2c66f23755cff08b8e27c9fd8fbf5ec688f4b00c66671d04"}
Oct 31 13:13:01.184730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:01.168384    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm" event={"ID":"525ea675-86c9-4900-b9c1-67bc9a203896","Type":"ContainerDied","Data":"c79abb3bbcc03546fe72973d94eab0c34ea11fc53daab87c6bae28be3e8a0c55"}
Oct 31 13:13:01.184730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:01.168447    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c79abb3bbcc03546fe72973d94eab0c34ea11fc53daab87c6bae28be3e8a0c55"
Oct 31 13:13:01.184730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:01.168522    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"
Oct 31 13:13:01.894914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:01.894878    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8h4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:13:01.895543 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:01.895507    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" podUID="80d56747-87ac-4de5-a195-7d50afcee740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:13:01.905619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:01.905572    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-72d7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 13:13:01.905770 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:01.905649    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podUID="2b850a6e-358a-4baa-8730-9b569d126952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 13:13:02.189678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:02.189631    5787 generic.go:356] "Generic (PLEG): container finished" podID="7965ee70-9bad-459e-8709-ac3a1d3cbf8f" containerID="adda22546f1bfb77e2bdd69e2c3df04199c590c63f826e821ac88cb9570d83ea" exitCode=0
Oct 31 13:13:02.189830 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:02.189690    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s" event={"ID":"7965ee70-9bad-459e-8709-ac3a1d3cbf8f","Type":"ContainerDied","Data":"adda22546f1bfb77e2bdd69e2c3df04199c590c63f826e821ac88cb9570d83ea"}
Oct 31 13:13:02.871521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:02.871481    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8h4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:13:02.871745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:02.871540    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" podUID="80d56747-87ac-4de5-a195-7d50afcee740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:13:02.905496 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:02.905455    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-72d7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 13:13:02.905898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:02.905521    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podUID="2b850a6e-358a-4baa-8730-9b569d126952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 13:13:03.419667 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.419633    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:03.476262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.476166    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-openshift-ca\") pod \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") "
Oct 31 13:13:03.476422 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.476389    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-xks-ca\") pod \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") "
Oct 31 13:13:03.476481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.476429    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-kube-api-access-frw8h\" (UniqueName: \"kubernetes.io/projected/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-kube-api-access-frw8h\") pod \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\" (UID: \"7965ee70-9bad-459e-8709-ac3a1d3cbf8f\") "
Oct 31 13:13:03.476662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.476634    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-openshift-ca" pod "7965ee70-9bad-459e-8709-ac3a1d3cbf8f" (UID: "7965ee70-9bad-459e-8709-ac3a1d3cbf8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:13:03.477196 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.477171    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:03.483121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.483092    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-xks-ca" pod "7965ee70-9bad-459e-8709-ac3a1d3cbf8f" (UID: "7965ee70-9bad-459e-8709-ac3a1d3cbf8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:13:03.483241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.483185    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-kube-api-access-frw8h" pod "7965ee70-9bad-459e-8709-ac3a1d3cbf8f" (UID: "7965ee70-9bad-459e-8709-ac3a1d3cbf8f"). InnerVolumeSpecName "kube-api-access-frw8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:13:03.578860 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.578820    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:03.578860 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.578854    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frw8h\" (UniqueName: \"kubernetes.io/projected/7965ee70-9bad-459e-8709-ac3a1d3cbf8f-kube-api-access-frw8h\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:03.871764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.871674    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8h4dd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 31 13:13:03.871914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.871761    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" podUID="80d56747-87ac-4de5-a195-7d50afcee740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 31 13:13:03.905665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.905629    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-72d7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 31 13:13:03.906077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:03.905741    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podUID="2b850a6e-358a-4baa-8730-9b569d126952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 31 13:13:04.215113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:04.209485    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s" event={"ID":"7965ee70-9bad-459e-8709-ac3a1d3cbf8f","Type":"ContainerDied","Data":"081c986069746faa2c66f23755cff08b8e27c9fd8fbf5ec688f4b00c66671d04"}
Oct 31 13:13:04.215113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:04.209513    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"
Oct 31 13:13:04.215113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:04.209587    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="081c986069746faa2c66f23755cff08b8e27c9fd8fbf5ec688f4b00c66671d04"
Oct 31 13:13:04.316728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:04.316672    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:13:04.397347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:04.396766    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:13:05.239113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:05.239066    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:13:05.349241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:05.349209    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:13:05.350622 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:05.350593    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:13:05.351651 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:05.351631    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r"
Oct 31 13:13:05.500816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:05.490313    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:13:05.500816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:05.491760    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:13:05.500816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:05.492359    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd"
Oct 31 13:13:06.847863 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.847826    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"]
Oct 31 13:13:06.848599 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.848574    5787 cpu_manager.go:414] "containerMap: removing container" podUID="7965ee70-9bad-459e-8709-ac3a1d3cbf8f" containerName="revoke-keys"
Oct 31 13:13:06.848678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.848600    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7965ee70-9bad-459e-8709-ac3a1d3cbf8f" containerName="revoke-keys"
Oct 31 13:13:06.848678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.848659    5787 cpu_manager.go:414] "containerMap: removing container" podUID="525ea675-86c9-4900-b9c1-67bc9a203896" containerName="revoke-keys"
Oct 31 13:13:06.848678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.848668    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="525ea675-86c9-4900-b9c1-67bc9a203896" containerName="revoke-keys"
Oct 31 13:13:06.848870 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.848855    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="7965ee70-9bad-459e-8709-ac3a1d3cbf8f" containerName="revoke-keys"
Oct 31 13:13:06.848912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.848884    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="525ea675-86c9-4900-b9c1-67bc9a203896" containerName="revoke-keys"
Oct 31 13:13:06.853049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.853025    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.863363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.863308    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"]
Oct 31 13:13:06.864958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.864934    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a2c006\"/\"openshift-service-ca.crt\""
Oct 31 13:13:06.865133 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.865013    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a2c006\"/\"kube-root-ca.crt\""
Oct 31 13:13:06.865266 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.865177    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a2c006\"/\"distinct-w1-a2c006-kserve-self-signed-certs\""
Oct 31 13:13:06.865266 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.865209    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a2c006\"/\"default-dockercfg-5bvlg\""
Oct 31 13:13:06.875790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.875756    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-home\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.876064 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.876031    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67r57\" (UniqueName: \"kubernetes.io/projected/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-kube-api-access-67r57\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.876193 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.876087    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tmp-dir\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.876193 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.876159    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tls-certs\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.876303 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.876229    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-dshm\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.876303 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.876245    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-model-cache\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.977799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.977679    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tls-certs\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.978170 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.978146    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-dshm\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.978399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.978381    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-model-cache\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.979064 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.979047    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-home\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.979540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.979522    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67r57\" (UniqueName: \"kubernetes.io/projected/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-kube-api-access-67r57\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.980093 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.980075    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tmp-dir\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.980668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.978944    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-model-cache\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.980827 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.979406    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-home\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.980953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.980484    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tmp-dir\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.997804 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.997417    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-dshm\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:06.998283 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:06.998232    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tls-certs\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:07.014627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:07.014546    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67r57\" (UniqueName: \"kubernetes.io/projected/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-kube-api-access-67r57\") pod \"distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:07.169874 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:07.169769    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:07.464916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:07.464865    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"]
Oct 31 13:13:07.465468 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:07.465445    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc65d5f1f_befb_4c32_8f0e_0a09f4915b53.slice/crio-d90baf6f74d681c91a9ad5b19a11a022f3df4093124ab2b3468ce9f37a891da2 WatchSource:0}: Error finding container d90baf6f74d681c91a9ad5b19a11a022f3df4093124ab2b3468ce9f37a891da2: Status 404 returned error can't find the container with id d90baf6f74d681c91a9ad5b19a11a022f3df4093124ab2b3468ce9f37a891da2
Oct 31 13:13:08.251782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:08.251746    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" event={"ID":"c65d5f1f-befb-4c32-8f0e-0a09f4915b53","Type":"ContainerStarted","Data":"8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7"}
Oct 31 13:13:08.251782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:08.251796    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" event={"ID":"c65d5f1f-befb-4c32-8f0e-0a09f4915b53","Type":"ContainerStarted","Data":"d90baf6f74d681c91a9ad5b19a11a022f3df4093124ab2b3468ce9f37a891da2"}
Oct 31 13:13:08.275562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:08.275505    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podStartSLOduration=2.275490361 podStartE2EDuration="2.275490361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:08.272945007 +0000 UTC m=+767.688886832" watchObservedRunningTime="2026-10-31 13:13:08.275490361 +0000 UTC m=+767.691432200"
Oct 31 13:13:09.169948 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:09.169909    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:10.676082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.676049    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"]
Oct 31 13:13:10.681721 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.681681    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:10.698756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.692458    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:13:10.698756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.695877    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:13:10.698756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.695993    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:13:10.698756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.696020    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:13:10.800504 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.800470    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sx2n\" (UniqueName: \"kubernetes.io/projected/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-kube-api-access-5sx2n\") pod \"maas-api-revoke-sub-5c4b8cef-sbxx7\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:10.800856 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.800837    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c4b8cef-sbxx7\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:10.801102 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.801085    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c4b8cef-sbxx7\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:10.803940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.803912    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"]
Oct 31 13:13:10.902190 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.902145    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c4b8cef-sbxx7\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:10.902377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.902214    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c4b8cef-sbxx7\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:10.902377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.902328    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sx2n\" (UniqueName: \"kubernetes.io/projected/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-kube-api-access-5sx2n\") pod \"maas-api-revoke-sub-5c4b8cef-sbxx7\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:10.902999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.902971    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c4b8cef-sbxx7\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:10.904951 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.904925    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c4b8cef-sbxx7\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:10.916430 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:10.916392    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sx2n\" (UniqueName: \"kubernetes.io/projected/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-kube-api-access-5sx2n\") pod \"maas-api-revoke-sub-5c4b8cef-sbxx7\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:11.110150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:11.110110    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:11.396353 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:11.396323    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"]
Oct 31 13:13:11.397388 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:11.397363    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71ff155_9ce4_4f6b_892a_3a17dd7fe837.slice/crio-ddf387ffefc97d944a925a828f545a517824259c396ca4fe367f05f5a6acf54f WatchSource:0}: Error finding container ddf387ffefc97d944a925a828f545a517824259c396ca4fe367f05f5a6acf54f: Status 404 returned error can't find the container with id ddf387ffefc97d944a925a828f545a517824259c396ca4fe367f05f5a6acf54f
Oct 31 13:13:12.306261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:12.306222    5787 generic.go:356] "Generic (PLEG): container finished" podID="b71ff155-9ce4-4f6b-892a-3a17dd7fe837" containerID="fc5b3c28c7847c80a435c528edcff916248761f6cbd641560e14ecf0e7a0e92c" exitCode=0
Oct 31 13:13:12.306742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:12.306282    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7" event={"ID":"b71ff155-9ce4-4f6b-892a-3a17dd7fe837","Type":"ContainerDied","Data":"fc5b3c28c7847c80a435c528edcff916248761f6cbd641560e14ecf0e7a0e92c"}
Oct 31 13:13:12.306742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:12.306337    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7" event={"ID":"b71ff155-9ce4-4f6b-892a-3a17dd7fe837","Type":"ContainerStarted","Data":"ddf387ffefc97d944a925a828f545a517824259c396ca4fe367f05f5a6acf54f"}
Oct 31 13:13:13.479292 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.479267    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:13.535263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.535220    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-kube-api-access-5sx2n\" (UniqueName: \"kubernetes.io/projected/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-kube-api-access-5sx2n\") pod \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") "
Oct 31 13:13:13.535443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.535285    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-xks-ca\") pod \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") "
Oct 31 13:13:13.535508 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.535475    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-openshift-ca\") pod \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\" (UID: \"b71ff155-9ce4-4f6b-892a-3a17dd7fe837\") "
Oct 31 13:13:13.535925 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.535892    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-openshift-ca" pod "b71ff155-9ce4-4f6b-892a-3a17dd7fe837" (UID: "b71ff155-9ce4-4f6b-892a-3a17dd7fe837"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:13:13.537669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.537646    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-xks-ca" pod "b71ff155-9ce4-4f6b-892a-3a17dd7fe837" (UID: "b71ff155-9ce4-4f6b-892a-3a17dd7fe837"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:13:13.538034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.538016    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-kube-api-access-5sx2n" pod "b71ff155-9ce4-4f6b-892a-3a17dd7fe837" (UID: "b71ff155-9ce4-4f6b-892a-3a17dd7fe837"). InnerVolumeSpecName "kube-api-access-5sx2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:13:13.637601 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.637508    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:13.637601 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.637538    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sx2n\" (UniqueName: \"kubernetes.io/projected/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-kube-api-access-5sx2n\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:13.637601 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:13.637552    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b71ff155-9ce4-4f6b-892a-3a17dd7fe837-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:13:14.218118 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.218075    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"]
Oct 31 13:13:14.218639 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.218622    5787 cpu_manager.go:414] "containerMap: removing container" podUID="b71ff155-9ce4-4f6b-892a-3a17dd7fe837" containerName="revoke-keys"
Oct 31 13:13:14.218765 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.218639    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71ff155-9ce4-4f6b-892a-3a17dd7fe837" containerName="revoke-keys"
Oct 31 13:13:14.218930 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.218907    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71ff155-9ce4-4f6b-892a-3a17dd7fe837" containerName="revoke-keys"
Oct 31 13:13:14.223139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.223107    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.233861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.232898    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-004f64\"/\"distinct-2-w3-004f64-kserve-self-signed-certs\""
Oct 31 13:13:14.236969 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.236935    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"]
Oct 31 13:13:14.246568 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.246528    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tmp-dir\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.246568 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.246577    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-dshm\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.246854 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.246608    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzxt8\" (UniqueName: \"kubernetes.io/projected/9c308e8e-d08e-43d5-accf-e9e9baa052b4-kube-api-access-wzxt8\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.246854 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.246662    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-model-cache\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.246854 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.246696    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-home\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.246854 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.246776    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tls-certs\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.333403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.333373    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"
Oct 31 13:13:14.333403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.333387    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7" event={"ID":"b71ff155-9ce4-4f6b-892a-3a17dd7fe837","Type":"ContainerDied","Data":"ddf387ffefc97d944a925a828f545a517824259c396ca4fe367f05f5a6acf54f"}
Oct 31 13:13:14.333654 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.333449    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddf387ffefc97d944a925a828f545a517824259c396ca4fe367f05f5a6acf54f"
Oct 31 13:13:14.351461 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.347953    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tls-certs\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.359261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.359225    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tmp-dir\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.365730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.359494    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-dshm\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.366000 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.365974    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzxt8\" (UniqueName: \"kubernetes.io/projected/9c308e8e-d08e-43d5-accf-e9e9baa052b4-kube-api-access-wzxt8\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.366150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.366134    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-model-cache\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.366378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.366349    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-home\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.375731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.366901    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-home\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.375978 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.361774    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tmp-dir\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.376119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.361427    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tls-certs\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.382730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.365401    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-dshm\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.382977 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.368162    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-model-cache\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.396732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.386504    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzxt8\" (UniqueName: \"kubernetes.io/projected/9c308e8e-d08e-43d5-accf-e9e9baa052b4-kube-api-access-wzxt8\") pod \"distinct-2-w3-004f64-kserve-5b948464c7-wv2bs\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") " pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.543163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.543017    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:14.846924 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:14.846894    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"]
Oct 31 13:13:14.847751 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:14.847692    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c308e8e_d08e_43d5_accf_e9e9baa052b4.slice/crio-ab1f25b62fadb31f6fec7be598dda6060e310fd7e9496f0fa5ea67a7dcf5334c WatchSource:0}: Error finding container ab1f25b62fadb31f6fec7be598dda6060e310fd7e9496f0fa5ea67a7dcf5334c: Status 404 returned error can't find the container with id ab1f25b62fadb31f6fec7be598dda6060e310fd7e9496f0fa5ea67a7dcf5334c
Oct 31 13:13:15.343243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:15.343206    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" event={"ID":"9c308e8e-d08e-43d5-accf-e9e9baa052b4","Type":"ContainerStarted","Data":"6f238a6bc40f38c95f0560700798de5483e2c9d11b29f26a0afaaad0787841db"}
Oct 31 13:13:15.343404 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:15.343257    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" event={"ID":"9c308e8e-d08e-43d5-accf-e9e9baa052b4","Type":"ContainerStarted","Data":"ab1f25b62fadb31f6fec7be598dda6060e310fd7e9496f0fa5ea67a7dcf5334c"}
Oct 31 13:13:15.367362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:15.367305    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podStartSLOduration=1.3672885940000001 podStartE2EDuration="1.367288594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:15.36484409 +0000 UTC m=+774.780785931" watchObservedRunningTime="2026-10-31 13:13:15.367288594 +0000 UTC m=+774.783230472"
Oct 31 13:13:15.545341 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:15.545280    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:17.170947 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:17.170907    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:17.184035 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:17.184005    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:17.376600 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:17.376572    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:18.248337 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:13:18.248300    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:13:21.649560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.649530    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"]
Oct 31 13:13:21.654560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.654540    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.674338 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.674303    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"]
Oct 31 13:13:21.723942 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.723901    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.724156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.723976    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.724156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.724086    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.724156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.724151    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.724349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.724177    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.724349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.724202    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1da508-b47c-4513-9584-a5b4771d4f5d-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.724458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.724369    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.724458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.724399    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.724561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.724467    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gncn7\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-kube-api-access-gncn7\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.826649    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.826736    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.826793    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gncn7\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-kube-api-access-gncn7\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.826833    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.826905    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.826949    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.827012    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.827036    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.827061    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1da508-b47c-4513-9584-a5b4771d4f5d-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.828010    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1da508-b47c-4513-9584-a5b4771d4f5d-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.828399    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.841105    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.842637    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.844729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.843160    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.850861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.850786    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.853251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.853220    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.858791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.858761    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.860847 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.860829    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gncn7\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-kube-api-access-gncn7\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:21.968762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:21.968726    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:22.009579 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.009544    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"]
Oct 31 13:13:22.015222 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.015195    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.056005 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.054455    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"]
Oct 31 13:13:22.130353 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.130320    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.130353 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.130357    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.130594 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.130456    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.130594 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.130502    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.130594 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.130529    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.130594 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.130544    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.130594 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.130588    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqpvk\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-kube-api-access-cqpvk\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.130871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.130664    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.130871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.130752    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.231548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.231455    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.231548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.231538    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.231812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.231591    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.231812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.231608    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.231812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.231647    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.231812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.231689    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.231812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.231743    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.231812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.231787    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.232191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.231833    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqpvk\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-kube-api-access-cqpvk\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.232191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.232111    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.232448 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.232421    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.232554 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.232430    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.232554 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.232461    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.232554 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.232464    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.235718 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.235685    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.235935 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.235909    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.251184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.251160    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.253784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.253759    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqpvk\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-kube-api-access-cqpvk\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.299227 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.299198    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"]
Oct 31 13:13:22.300131 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:22.300098    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1da508_b47c_4513_9584_a5b4771d4f5d.slice/crio-61b1b9a8d25af21576adae1ada13fc520c27fe1477f7e796df7ffe0357c8e34c WatchSource:0}: Error finding container 61b1b9a8d25af21576adae1ada13fc520c27fe1477f7e796df7ffe0357c8e34c: Status 404 returned error can't find the container with id 61b1b9a8d25af21576adae1ada13fc520c27fe1477f7e796df7ffe0357c8e34c
Oct 31 13:13:22.303940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.303900    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:22.304062 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.303972    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:22.304062 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.304001    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:22.336899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.336860    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:22.405374 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.404914    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" event={"ID":"8a1da508-b47c-4513-9584-a5b4771d4f5d","Type":"ContainerStarted","Data":"f1a4b404ccdaca53844fafb7a55b0fbbdfea74b1e136bc2fe97087b9742fd770"}
Oct 31 13:13:22.405374 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.404985    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" event={"ID":"8a1da508-b47c-4513-9584-a5b4771d4f5d","Type":"ContainerStarted","Data":"61b1b9a8d25af21576adae1ada13fc520c27fe1477f7e796df7ffe0357c8e34c"}
Oct 31 13:13:22.440901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.438380    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podStartSLOduration=1.438360889 podStartE2EDuration="1.438360889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:22.43435049 +0000 UTC m=+781.850292332" watchObservedRunningTime="2026-10-31 13:13:22.438360889 +0000 UTC m=+781.854302731"
Oct 31 13:13:22.683809 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.682561    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"]
Oct 31 13:13:22.733676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.733623    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:22.734007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.733959    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:22.734167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.734151    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:22.981730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:22.969398    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:23.477732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:23.477056    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" event={"ID":"365f64c7-a490-4ee0-87e1-56c68d22d5f3","Type":"ContainerStarted","Data":"4f0b8a6b8a7a12513e86be3f4dc02176003909f93c3f926f38d02f89f8ebc82a"}
Oct 31 13:13:23.477732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:23.477127    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" event={"ID":"365f64c7-a490-4ee0-87e1-56c68d22d5f3","Type":"ContainerStarted","Data":"e81a33b864ae6450af6caa4e28ffa7ebab8045c3bc16b5ef4014640b0eafb98d"}
Oct 31 13:13:23.973930 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:23.973885    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:13:23.974568 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:23.974527    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 13:13:24.347782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:24.342080    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:24.368832 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:24.348368    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:13:24.368832 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:24.348436    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:13:24.375744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:24.374101    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:24.552728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:24.552102    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:24.681796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:24.668883    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:24.970730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:24.970151    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:13:24.970730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:24.970215    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 13:13:25.126730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.125808    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:25.339781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.339589    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:13:25.339781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.339660    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:13:25.386728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.377889    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:25.386728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.378225    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:25.417721 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.408837    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podStartSLOduration=4.408815142 podStartE2EDuration="4.408815142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:23.534223663 +0000 UTC m=+782.950165503" watchObservedRunningTime="2026-10-31 13:13:25.408815142 +0000 UTC m=+784.824756983"
Oct 31 13:13:25.570731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.560841    5787 prober.go:122] "Probe failed" probeType="Startup" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded"
Oct 31 13:13:25.586825 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.579128    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:13:25.757731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.757178    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:25.757939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.757873    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:13:25.787766 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.782840    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:13:25.829731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.827933    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:25.831084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.830689    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:25.986152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.970112    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:13:25.986152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:25.983369    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 13:13:26.029250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:26.029130    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:13:26.341209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:26.341117    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:13:26.342226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:26.341812    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:13:26.974731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:26.974141    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:13:26.974731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:26.974206    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 13:13:27.342793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:27.340135    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:13:27.342793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:27.340199    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:13:27.978733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:27.978045    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:13:27.978733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:27.978142    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 13:13:28.338306 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:28.338201    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:13:28.338577 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:28.338541    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:13:28.970368 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:28.970332    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:13:28.971053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:28.970992    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 13:13:29.338208 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:29.338112    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:13:29.338463 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:29.338435    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:13:29.971440 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:29.971403    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 31 13:13:29.972052 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:29.972021    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 31 13:13:30.337410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:30.337299    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 31 13:13:30.337410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:30.337369    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 31 13:13:30.975936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:30.975900    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:30.976579 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:30.976553    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:30.977479 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:30.977457    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:13:31.342044 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:31.341967    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:31.342458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:31.342446    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:31.343252 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:31.343229    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:13:34.556270 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:34.556237    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:34.573919 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:34.573890    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:13:36.571110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.571075    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"]
Oct 31 13:13:36.576542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.576517    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.667994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.667959    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"]
Oct 31 13:13:36.690731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.668529    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2436534-3062-4a6d-ba08-93e324b1ec85-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.690731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.668640    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.690731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.668669    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.690731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.677217    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.690731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.677261    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.690731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.677289    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kv4\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-kube-api-access-79kv4\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.690731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.677360    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.690731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.677385    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.690731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.677489    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.784287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.784248    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.784548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.784525    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.784662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.784647    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kv4\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-kube-api-access-79kv4\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.784819 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.784805    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.784923 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.784910    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.785084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.785072    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.785172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.785161    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2436534-3062-4a6d-ba08-93e324b1ec85-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.787136 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.785335    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.789986 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.789956    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.791244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.791190    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.791792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.791754    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.796669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.796634    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.798182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.797729    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.799946 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.799924    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2436534-3062-4a6d-ba08-93e324b1ec85-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.803186 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.803163    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.839115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.838970    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kv4\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-kube-api-access-79kv4\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.841427 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.841395    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.844097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.844038    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:36.897566 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:36.897517    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:37.217377 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:37.217345    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2436534_3062_4a6d_ba08_93e324b1ec85.slice/crio-a75583a0bc0c4210524984178e97c03105a3178b18e891e0509c40f6eb1c3bc0 WatchSource:0}: Error finding container a75583a0bc0c4210524984178e97c03105a3178b18e891e0509c40f6eb1c3bc0: Status 404 returned error can't find the container with id a75583a0bc0c4210524984178e97c03105a3178b18e891e0509c40f6eb1c3bc0
Oct 31 13:13:37.233227 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.233201    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"]
Oct 31 13:13:37.237541 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.237503    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"]
Oct 31 13:13:37.242906 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.242851    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:37.243056 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.242956    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:37.243056 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.243004    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:37.271653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.271618    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"]
Oct 31 13:13:37.271856 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.271782    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.305215 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.305179    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.305380 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.305225    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f71d479-bb83-40b8-a668-9865152cf25d-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.305380 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.305250    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4662\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-kube-api-access-q4662\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.305511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.305481    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.305577 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.305541    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f71d479-bb83-40b8-a668-9865152cf25d-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.305627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.305578    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.305627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.305608    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.305825 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.305805    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.305950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.305934    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407462    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407524    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407601    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407632    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f71d479-bb83-40b8-a668-9865152cf25d-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407661    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4662\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-kube-api-access-q4662\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407797    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407838    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f71d479-bb83-40b8-a668-9865152cf25d-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407869    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407897    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.407973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.407949    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.408485 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.408462    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.408633 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.408495    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.408859 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.408837    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.408948 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.408875    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f71d479-bb83-40b8-a668-9865152cf25d-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.410560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.410536    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f71d479-bb83-40b8-a668-9865152cf25d-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.410946 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.410930    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.421936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.421660    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4662\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-kube-api-access-q4662\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.421936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.421881    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.589624 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.589542    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:37.760983 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.760943    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" event={"ID":"e2436534-3062-4a6d-ba08-93e324b1ec85","Type":"ContainerStarted","Data":"b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8"}
Oct 31 13:13:37.761244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.761011    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" event={"ID":"e2436534-3062-4a6d-ba08-93e324b1ec85","Type":"ContainerStarted","Data":"a75583a0bc0c4210524984178e97c03105a3178b18e891e0509c40f6eb1c3bc0"}
Oct 31 13:13:37.789989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.789939    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podStartSLOduration=1.789925964 podStartE2EDuration="1.789925964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:37.785047629 +0000 UTC m=+797.200989469" watchObservedRunningTime="2026-10-31 13:13:37.789925964 +0000 UTC m=+797.205867803"
Oct 31 13:13:37.897418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.897390    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"]
Oct 31 13:13:37.897569 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.897555    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:37.898040 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:37.898015    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f71d479_bb83_40b8_a668_9865152cf25d.slice/crio-c0fb77113c9475b87fc2f15c61bf5997ab2249f832c672b002d28a5c8f4dfc25 WatchSource:0}: Error finding container c0fb77113c9475b87fc2f15c61bf5997ab2249f832c672b002d28a5c8f4dfc25: Status 404 returned error can't find the container with id c0fb77113c9475b87fc2f15c61bf5997ab2249f832c672b002d28a5c8f4dfc25
Oct 31 13:13:37.900769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.900739    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:37.900841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.900802    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:37.900841 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:37.900832    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:38.772658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:38.772611    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" event={"ID":"9f71d479-bb83-40b8-a668-9865152cf25d","Type":"ContainerStarted","Data":"c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6"}
Oct 31 13:13:38.773156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:38.772759    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" event={"ID":"9f71d479-bb83-40b8-a668-9865152cf25d","Type":"ContainerStarted","Data":"c0fb77113c9475b87fc2f15c61bf5997ab2249f832c672b002d28a5c8f4dfc25"}
Oct 31 13:13:38.806445 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:38.806366    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podStartSLOduration=1.8063457490000001 podStartE2EDuration="1.806345749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:38.800137684 +0000 UTC m=+798.216079526" watchObservedRunningTime="2026-10-31 13:13:38.806345749 +0000 UTC m=+798.222287592"
Oct 31 13:13:38.899517 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:38.899479    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 13:13:38.899768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:38.899555    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 13:13:39.614630 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:39.614596    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:39.625240 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:39.624852    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:13:39.625240 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:39.624929    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:13:39.898791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:39.898166    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 13:13:39.898791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:39.898229    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 13:13:40.590107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:40.590068    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:13:40.590382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:40.590354    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:13:40.899957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:40.899855    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 13:13:40.900325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:40.899939    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 13:13:41.590167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:41.590127    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:13:41.590474 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:41.590447    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:13:41.898411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:41.898300    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 13:13:41.898411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:41.898377    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 13:13:42.590738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:42.590150    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:13:42.590738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:42.590216    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:13:42.900101 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:42.899963    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 13:13:42.900101 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:42.900021    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 13:13:43.192855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.190465    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"]
Oct 31 13:13:43.195831 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.195802    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.213894 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.213865    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a2c006\"/\"distinct-2-w1-a2c006-kserve-self-signed-certs\""
Oct 31 13:13:43.232808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.232766    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"]
Oct 31 13:13:43.245737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.236645    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-home\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.245737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.236749    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tmp-dir\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.245737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.236869    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpcqz\" (UniqueName: \"kubernetes.io/projected/bf8cdef9-566f-443d-9949-3a8e9947fd4e-kube-api-access-mpcqz\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.245737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.236972    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-model-cache\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.245737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.237038    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tls-certs\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.245737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.237093    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-dshm\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.355756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.355509    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-model-cache\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.355756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.355617    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tls-certs\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.355756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.355676    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-dshm\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.356057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.355768    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-home\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.356057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.355826    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tmp-dir\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.356057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.355941    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpcqz\" (UniqueName: \"kubernetes.io/projected/bf8cdef9-566f-443d-9949-3a8e9947fd4e-kube-api-access-mpcqz\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.362734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.356692    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-model-cache\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.362903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.362860    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tls-certs\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.377736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.376184    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-home\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.377736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.376525    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tmp-dir\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.399291 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.394185    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-dshm\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.437727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.423827    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpcqz\" (UniqueName: \"kubernetes.io/projected/bf8cdef9-566f-443d-9949-3a8e9947fd4e-kube-api-access-mpcqz\") pod \"distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") " pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.533781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.532986    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:43.618498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.618458    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:13:43.619150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.619100    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:13:43.901665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.901270    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 13:13:43.901665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:43.901332    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 13:13:44.196681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:44.196650    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"]
Oct 31 13:13:44.196900 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:44.196863    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf8cdef9_566f_443d_9949_3a8e9947fd4e.slice/crio-20dcc84f7eb3cf1b6e82dfd6d7e3cbdc563bb67a89844d86f3b70f414e270306 WatchSource:0}: Error finding container 20dcc84f7eb3cf1b6e82dfd6d7e3cbdc563bb67a89844d86f3b70f414e270306: Status 404 returned error can't find the container with id 20dcc84f7eb3cf1b6e82dfd6d7e3cbdc563bb67a89844d86f3b70f414e270306
Oct 31 13:13:44.590788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:44.589946    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body=
Oct 31 13:13:44.590788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:44.589994    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused"
Oct 31 13:13:45.083729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:45.078791    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" event={"ID":"bf8cdef9-566f-443d-9949-3a8e9947fd4e","Type":"ContainerStarted","Data":"13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb"}
Oct 31 13:13:45.083729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:45.078844    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" event={"ID":"bf8cdef9-566f-443d-9949-3a8e9947fd4e","Type":"ContainerStarted","Data":"20dcc84f7eb3cf1b6e82dfd6d7e3cbdc563bb67a89844d86f3b70f414e270306"}
Oct 31 13:13:45.117734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:45.109059    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podStartSLOduration=2.109034479 podStartE2EDuration="2.109034479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:45.104446177 +0000 UTC m=+804.520388018" watchObservedRunningTime="2026-10-31 13:13:45.109034479 +0000 UTC m=+804.524976320"
Oct 31 13:13:45.386775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:45.385829    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:45.386775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:45.386296    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:45.391729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:45.387300    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:13:45.533855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:45.533808    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:46.071588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:46.071546    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:46.071997 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:46.071929    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:46.072811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:46.072794    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:13:48.456698 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:13:48.456651    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:13:51.562432 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.562388    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"]
Oct 31 13:13:51.567472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.567429    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.583306 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.583225    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"]
Oct 31 13:13:51.752428 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.752392    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.752646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.752579    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.752733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.752692    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.752793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.752732    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94pkd\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-kube-api-access-94pkd\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.752793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.752753    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.752910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.752884    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.752994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.752979    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.753070 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.753057    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.753123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.753089    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.854359 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.854267    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.854507 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.854371    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.854507 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.854398    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94pkd\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-kube-api-access-94pkd\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.854507 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.854427    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.854507 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.854474    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.854685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.854568    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.854685 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.854652    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.855138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.855112    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.855138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.855128    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.855411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.855381    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.855497 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.855432    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.855497 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.855466    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.855788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.855768    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.856331 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.856307    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.857251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.857225    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.859105 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.859081    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.883267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.883230    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.884797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.884767    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94pkd\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-kube-api-access-94pkd\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.887573 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.887549    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:51.900984 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.900560    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"]
Oct 31 13:13:51.905018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.904995    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:51.914921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.914892    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a2c006-openshift-default-dockercfg-shg42\""
Oct 31 13:13:51.920756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.920722    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"]
Oct 31 13:13:51.971452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.969621    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527f4c39-43e6-4506-b95d-42423d24c688-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:51.971452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.969690    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:51.971452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.969820    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:51.971452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.969852    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:51.971452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.969913    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527f4c39-43e6-4506-b95d-42423d24c688-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:51.971452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.969938    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:51.971452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.969973    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xtb\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-kube-api-access-f6xtb\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:51.971452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.970052    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:51.971452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:51.970146    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.077643    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.077719    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.077772    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527f4c39-43e6-4506-b95d-42423d24c688-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.077798    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.077835    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xtb\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-kube-api-access-f6xtb\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.077926    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.078006    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.078092    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527f4c39-43e6-4506-b95d-42423d24c688-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.078138    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.078730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.078595    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.079414 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.078948    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.079414 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.079178    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.120000 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.105452    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.120000 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.113892    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527f4c39-43e6-4506-b95d-42423d24c688-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.129738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.123441    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"]
Oct 31 13:13:52.130665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.130630    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xtb\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-kube-api-access-f6xtb\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.130959 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.130939    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.131162 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.131101    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.132307 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.132281    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.134447 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.134420    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527f4c39-43e6-4506-b95d-42423d24c688-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.145914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.145805    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"]
Oct 31 13:13:52.160690 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.160654    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"]
Oct 31 13:13:52.166912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.166884    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.181449 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.181364    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"]
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186324    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186379    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186420    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186443    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186472    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186498    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrk5g\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-kube-api-access-qrk5g\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186535    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186572    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kz28\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-kube-api-access-2kz28\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186634    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186657    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186751    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.186972    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.187036    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.187067    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd55913-612c-4bc3-a067-d11002f9f182-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.187105    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.187129    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.187167    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.188389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.187195    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd55913-612c-4bc3-a067-d11002f9f182-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.204589 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.204565    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"]
Oct 31 13:13:52.206801 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:52.206776    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e11ad3c_692f_4245_a6d8_68b4f945b3d1.slice/crio-df76726b436f4e81bf0d94c3c68b6830aadd1c1473e405b398ad04aaa79aca69 WatchSource:0}: Error finding container df76726b436f4e81bf0d94c3c68b6830aadd1c1473e405b398ad04aaa79aca69: Status 404 returned error can't find the container with id df76726b436f4e81bf0d94c3c68b6830aadd1c1473e405b398ad04aaa79aca69
Oct 31 13:13:52.209970 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.209933    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.210085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.210021    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.210085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.210067    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.256347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.256200    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"]
Oct 31 13:13:52.259064 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.259036    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:52.262466 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.262444    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.278087 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.277999    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"]
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303590    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303657    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303697    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303754    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303781    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303805    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303834    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303859    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrk5g\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-kube-api-access-qrk5g\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303897    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303935    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kz28\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-kube-api-access-2kz28\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303972    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.303998    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304058    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304084    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304151    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304245    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304271    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-kube-api-access-nfl2f\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304367    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304404    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304446    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304484    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304516    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd55913-612c-4bc3-a067-d11002f9f182-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304554    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304579    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304618    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.305598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304643    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd55913-612c-4bc3-a067-d11002f9f182-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.304668    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.306203    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.306451    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.306657    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.307006    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.307495    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-credential-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.307895    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-socket\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.308787    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-certs\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.309402    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.309619    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-data\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.310957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.310442    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd55913-612c-4bc3-a067-d11002f9f182-istiod-ca-cert\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.313365 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.313286    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.313365 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.313304    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-envoy\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.314031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.313989    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.315416 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.315373    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd55913-612c-4bc3-a067-d11002f9f182-istio-podinfo\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.319641 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.319310    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.321948 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.321493    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrk5g\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-kube-api-access-qrk5g\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.324637 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.324583    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-istio-token\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.333458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.333331    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kz28\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-kube-api-access-2kz28\") pod \"e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") " pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.412433    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.412494    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.412553    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.412640    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.412670    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.412743    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.412771    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.412902    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.412925    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-kube-api-access-nfl2f\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.413348    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.413597    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.413856    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.414938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.414200    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.415640 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.415616    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.416996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.416528    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.416996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.416957    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.428336 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.428278    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.431091 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.431061    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-kube-api-access-nfl2f\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.449869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.449838    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:52.493700 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.493604    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:52.585623 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.585397    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:52.596341 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.596273    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"]
Oct 31 13:13:52.605333 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.605287    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.605467 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.605381    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.605467 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.605423    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.759352 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.757493    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"]
Oct 31 13:13:52.764165 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:52.761692    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f8f023_3eb8_429c_8a06_d9a39fe51981.slice/crio-a18e3946a4125b231270b497c9970fdbfdb2450a72409a6fd204552b80d52814 WatchSource:0}: Error finding container a18e3946a4125b231270b497c9970fdbfdb2450a72409a6fd204552b80d52814: Status 404 returned error can't find the container with id a18e3946a4125b231270b497c9970fdbfdb2450a72409a6fd204552b80d52814
Oct 31 13:13:52.767226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.767187    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.767347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.767279    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.767347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.767316    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.822852 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.822823    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"]
Oct 31 13:13:52.845834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.845791    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.846118 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.846090    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.846307 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.846291    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:52.982991 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:52.982108    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"]
Oct 31 13:13:52.983173 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:13:52.983058    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc4b5f7e_1bed_46dd_b27f_e89f250107ec.slice/crio-5ce283cff5ed32fb71954bd077319bfeca4c71ea5c733453145f31dc1b616ba3 WatchSource:0}: Error finding container 5ce283cff5ed32fb71954bd077319bfeca4c71ea5c733453145f31dc1b616ba3: Status 404 returned error can't find the container with id 5ce283cff5ed32fb71954bd077319bfeca4c71ea5c733453145f31dc1b616ba3
Oct 31 13:13:53.023608 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.023547    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:53.023915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.023889    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:53.024154 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.024133    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:13:53.197775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.188992    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" event={"ID":"527f4c39-43e6-4506-b95d-42423d24c688","Type":"ContainerStarted","Data":"b740bbdcd7a0954163d955da68a69783d18600fef1a192a6b97535cf2924a51b"}
Oct 31 13:13:53.197775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.189061    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" event={"ID":"527f4c39-43e6-4506-b95d-42423d24c688","Type":"ContainerStarted","Data":"21d87e7c8391bf2f784ef38c6291af6cbaa4bc6874d18542cf235fd79e65d4f3"}
Oct 31 13:13:53.224822 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.224784    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" event={"ID":"c6f8f023-3eb8-429c-8a06-d9a39fe51981","Type":"ContainerStarted","Data":"49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16"}
Oct 31 13:13:53.225095 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.225069    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" event={"ID":"c6f8f023-3eb8-429c-8a06-d9a39fe51981","Type":"ContainerStarted","Data":"a18e3946a4125b231270b497c9970fdbfdb2450a72409a6fd204552b80d52814"}
Oct 31 13:13:53.238298 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.236235    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" event={"ID":"4e11ad3c-692f-4245-a6d8-68b4f945b3d1","Type":"ContainerStarted","Data":"d96380fb7e94a3a3cae82661f8384687866d8dfbdc6349fc055241f10bd2d939"}
Oct 31 13:13:53.238298 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.236292    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" event={"ID":"4e11ad3c-692f-4245-a6d8-68b4f945b3d1","Type":"ContainerStarted","Data":"df76726b436f4e81bf0d94c3c68b6830aadd1c1473e405b398ad04aaa79aca69"}
Oct 31 13:13:53.242041 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.241211    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" event={"ID":"bc4b5f7e-1bed-46dd-b27f-e89f250107ec","Type":"ContainerStarted","Data":"82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b"}
Oct 31 13:13:53.242041 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.241267    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" event={"ID":"bc4b5f7e-1bed-46dd-b27f-e89f250107ec","Type":"ContainerStarted","Data":"5ce283cff5ed32fb71954bd077319bfeca4c71ea5c733453145f31dc1b616ba3"}
Oct 31 13:13:53.244785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.244755    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" event={"ID":"abd55913-612c-4bc3-a067-d11002f9f182","Type":"ContainerStarted","Data":"f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9"}
Oct 31 13:13:53.245787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.245003    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" event={"ID":"abd55913-612c-4bc3-a067-d11002f9f182","Type":"ContainerStarted","Data":"dcf6a05d732117ae0ab1d72e90e76520a10d8e0e5e5d9a04737b4b7b9a2c422f"}
Oct 31 13:13:53.246658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.246609    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podStartSLOduration=2.246589727 podStartE2EDuration="2.246589727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:53.231586473 +0000 UTC m=+812.647528317" watchObservedRunningTime="2026-10-31 13:13:53.246589727 +0000 UTC m=+812.662531567"
Oct 31 13:13:53.261876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.261846    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:13:53.262139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.262121    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:13:53.262258 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.262236    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:13:53.341035 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.340971    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podStartSLOduration=1.340948365 podStartE2EDuration="1.340948365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:53.262833495 +0000 UTC m=+812.678775338" watchObservedRunningTime="2026-10-31 13:13:53.340948365 +0000 UTC m=+812.756890208"
Oct 31 13:13:53.390758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.387297    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podStartSLOduration=2.387276841 podStartE2EDuration="2.387276841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:53.340550141 +0000 UTC m=+812.756491983" watchObservedRunningTime="2026-10-31 13:13:53.387276841 +0000 UTC m=+812.803218682"
Oct 31 13:13:53.390758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.389346    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podStartSLOduration=1.3893312660000001 podStartE2EDuration="1.389331266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:53.38168819 +0000 UTC m=+812.797630030" watchObservedRunningTime="2026-10-31 13:13:53.389331266 +0000 UTC m=+812.805273107"
Oct 31 13:13:53.414632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.414543    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podStartSLOduration=1.41452136 podStartE2EDuration="1.41452136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:13:53.413634282 +0000 UTC m=+812.829576123" watchObservedRunningTime="2026-10-31 13:13:53.41452136 +0000 UTC m=+812.830463201"
Oct 31 13:13:53.466829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.460050    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:13:53.466829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.460232    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:13:53.466829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.460278    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:13:53.494367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.494329    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:13:53.537732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.533733    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:53.586849 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.586808    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:13:53.598026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.597989    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:53.894635 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.887961    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:13:53.898229 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.898196    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:13:53.898464 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:53.898438    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:13:54.293730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.286955    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:13:54.293730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.287042    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:13:54.461731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.457896    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:13:54.461731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.457953    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:13:54.497070 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.496679    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:13:54.497070 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.496767    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:13:54.594784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.594344    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:13:54.594784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.594404    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:13:54.651528 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.650805    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:13:54.713831 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.713785    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:54.758730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.755815    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:54.800732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.799791    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:54.821960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.821919    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:54.889230 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.889136    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:13:54.889478 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:54.889452    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:13:55.033735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.032818    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:55.033735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.033207    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:13:55.123861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.123809    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:55.232730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.228800    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:13:55.264728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.261108    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:13:55.264728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.261171    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:13:55.306998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.296235    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:55.324940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.324893    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:55.449986 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.448834    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:55.450733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.450508    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:13:55.450733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.450563    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:13:55.574884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.574793    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:13:55.575148 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.575117    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:13:55.599830 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.599795    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:13:55.626851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.626808    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:13:55.695560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.621232    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:55.697308 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.695573    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:55.697967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.697821    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:13:55.936847 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.928473    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:13:55.937525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:55.937128    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:13:56.048364 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.048299    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:56.049116 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.049094    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:13:56.233793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.229800    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:56.233793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.230430    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:13:56.492811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.463861    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:13:56.492811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.463925    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:13:56.494247 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.494217    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:13:56.494490 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.494453    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:13:56.587739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.556837    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:13:56.587739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.556910    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:13:56.590541 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.590510    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:13:56.590850 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.590818    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:13:56.741380 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.726434    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:56.796113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.796007    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:56.803334 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.803261    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:13:56.838758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.830484    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:56.839047 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.813904    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:56.847962 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.846858    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:56.866131 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.866095    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:13:56.870915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.870832    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:56.907959 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.907922    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:13:56.908849 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:56.908815    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:13:57.047045 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.037022    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"]
Oct 31 13:13:57.065974 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.055655    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:13:57.109733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.099949    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:13:57.109733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.099994    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"]
Oct 31 13:13:57.109733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.100101    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.109733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.107026    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:13:57.110966 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.110941    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:13:57.111212 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.111194    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:13:57.111458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.111439    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:13:57.246757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.219860    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23d31bd-j77zp\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.276731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.272969    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sssl\" (UniqueName: \"kubernetes.io/projected/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-kube-api-access-7sssl\") pod \"maas-api-revoke-sub-b23d31bd-j77zp\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.282730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.279159    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23d31bd-j77zp\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.286153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.247069    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:13:57.321784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.321648    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:13:57.335745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.335276    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:13:57.335745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.335332    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:13:57.403758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.385003    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23d31bd-j77zp\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.404525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.404499    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sssl\" (UniqueName: \"kubernetes.io/projected/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-kube-api-access-7sssl\") pod \"maas-api-revoke-sub-b23d31bd-j77zp\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.408949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.408920    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23d31bd-j77zp\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.414047 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.409975    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b23d31bd-j77zp\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.441280 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.441241    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b23d31bd-j77zp\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.491731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.487817    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:57.506733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.503143    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:13:57.506970 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.506748    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:13:57.521732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.518861    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:13:57.521732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.518927    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:13:57.567784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.562817    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:57.602791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.598538    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:13:57.602791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.602436    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:13:57.602791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.602482    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:13:57.705755 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.705722    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sssl\" (UniqueName: \"kubernetes.io/projected/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-kube-api-access-7sssl\") pod \"maas-api-revoke-sub-b23d31bd-j77zp\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:57.797012 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.796961    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:57.798318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.798294    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:13:57.838535 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.836800    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:57.838535 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.837482    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:13:57.842316 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.840790    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:57.842316 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.841153    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:13:57.856842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.856619    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:57.882193 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.882126    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:57.886368 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.886340    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:13:57.899739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.898229    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:13:57.899739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.898309    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:13:57.939522 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.939479    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:13:57.970861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:57.958476    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:13:58.088902 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.087076    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:13:58.088902 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.087816    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:58.115588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.107132    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:58.260742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.260678    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:58.310853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.310815    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:13:58.311130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.311096    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:13:58.312815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.312786    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:13:58.329738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.327123    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:13:58.329738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.327580    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:13:58.459733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.454226    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:13:58.459733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.454287    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:13:58.498223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.498182    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:13:58.498465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.498437    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:13:58.587229 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.587194    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:13:58.587499 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.587467    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:13:58.816787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.804404    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:58.841196 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.841152    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:58.864661 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.863539    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:58.882666 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.882628    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:58.897744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.892432    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:13:58.897744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.892487    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:13:58.897744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:58.892955    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:13:59.038734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.033334    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"]
Oct 31 13:13:59.105100 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.093756    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:59.119736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.111856    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:59.119736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.112308    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:13:59.263732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.262853    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:13:59.263732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.262919    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:13:59.318992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.318950    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:13:59.332521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.332382    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:59.335268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.334231    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:13:59.471261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.471228    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:13:59.471511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.471481    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:13:59.480519 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.472463    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:13:59.495399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.495367    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:13:59.508609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.508566    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:13:59.624767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.623743    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:59.649785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.625131    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:13:59.649785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.625182    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:13:59.791764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.783207    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp" event={"ID":"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4","Type":"ContainerStarted","Data":"5471e4a5dc57e5aaedabfbce269a1bdf23a215c85bff07eb2f0f4c497bb5fca5"}
Oct 31 13:13:59.824174 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.824120    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 31 13:13:59.872215 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.872153    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:59.890934 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.890876    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:59.911920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.907414    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:13:59.955426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.950604    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:13:59.955426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.951410    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:13:59.955426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:13:59.951450    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:14:00.122742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.121933    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:00.122742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.122666    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:00.224742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.207298    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:14:00.264697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.264273    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:14:00.264697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.264335    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:14:00.395224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.381034    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"]
Oct 31 13:14:00.411044 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.411014    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:00.453769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.452928    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:14:00.453769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.452984    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:14:00.469240 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.469203    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"]
Oct 31 13:14:00.503326 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.503232    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef56e0e2-f728k\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:00.504072 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.504049    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8llm\" (UniqueName: \"kubernetes.io/projected/841f919b-9011-4970-98aa-2d0cb8fe36f6-kube-api-access-r8llm\") pod \"maas-api-revoke-sub-ef56e0e2-f728k\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:00.505292 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.505271    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef56e0e2-f728k\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:00.505531 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.503891    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:14:00.505762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.505738    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:14:00.599757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.596799    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:00.631829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.629888    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:00.631829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.630338    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:00.797786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.780255    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:14:00.797786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.780318    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:14:00.849813 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.849589    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 31 13:14:00.873192 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.873155    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:00.918726 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.915432    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 31 13:14:00.921698 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.921655    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 31 13:14:01.075392 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.979906    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:01.100561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:00.979962    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:01.100826 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.043514    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8llm\" (UniqueName: \"kubernetes.io/projected/841f919b-9011-4970-98aa-2d0cb8fe36f6-kube-api-access-r8llm\") pod \"maas-api-revoke-sub-ef56e0e2-f728k\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:01.101148 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.101126    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef56e0e2-f728k\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:01.101440 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.101422    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef56e0e2-f728k\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:01.102668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.102644    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef56e0e2-f728k\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:01.125357 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.125317    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:01.190829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.126181    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:01.192340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.161541    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:14:01.192523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.192496    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:14:01.236764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.208208    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp" podStartSLOduration=5.208186324 podStartE2EDuration="5.208186324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:01.205657859 +0000 UTC m=+820.621599700" watchObservedRunningTime="2026-10-31 13:14:01.208186324 +0000 UTC m=+820.624128164"
Oct 31 13:14:01.583732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.581492    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef56e0e2-f728k\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:01.583732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.581920    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp" event={"ID":"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4","Type":"ContainerStarted","Data":"a85ecf9804f6d9baa0493f769d8f30c8027fa314822463e013f412297813d408"}
Oct 31 13:14:01.583732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.582046    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:14:01.601732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.588887    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8llm\" (UniqueName: \"kubernetes.io/projected/841f919b-9011-4970-98aa-2d0cb8fe36f6-kube-api-access-r8llm\") pod \"maas-api-revoke-sub-ef56e0e2-f728k\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:01.602246 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.602204    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:01.615261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.615227    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:01.618260 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.618230    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:14:01.618538 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.618498    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:14:01.647730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.644265    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:14:01.647730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.645649    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:14:01.647730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.645729    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:14:01.654762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.653173    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:14:01.654762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.653230    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:14:01.660741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.659625    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:14:01.660741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.659672    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:14:01.669001 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.661986    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:01.735297 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.727353    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"]
Oct 31 13:14:01.942605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.898479    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:14:01.942605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.942046    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:14:01.942605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.919892    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:01.955847 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.920489    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:01.964813 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:01.909874    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:02.081186 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.081158    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"]
Oct 31 13:14:02.199855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.153026    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:02.199855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.181060    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:02.199855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.182458    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:02.199855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.182540    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:02.199855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.193751    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:02.199855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.194050    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:02.211960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.200689    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:14:02.223126 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.223086    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aea6021-b24xb\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:02.243878 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.243838    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvpxl\" (UniqueName: \"kubernetes.io/projected/641a74ef-ffdc-417d-86be-b38c42d67de3-kube-api-access-gvpxl\") pod \"maas-api-revoke-sub-3aea6021-b24xb\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:02.244185 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.244155    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aea6021-b24xb\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:02.245207 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.245176    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:02.286817 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.286677    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:14:02.286817 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.286755    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:14:02.367795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.366237    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvpxl\" (UniqueName: \"kubernetes.io/projected/641a74ef-ffdc-417d-86be-b38c42d67de3-kube-api-access-gvpxl\") pod \"maas-api-revoke-sub-3aea6021-b24xb\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:02.367795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.366321    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aea6021-b24xb\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:02.367795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.366488    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aea6021-b24xb\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:02.412762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.402836    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aea6021-b24xb\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:02.425101 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.425063    5787 generic.go:356] "Generic (PLEG): container finished" podID="a496c77d-33bb-4c88-8de7-1ffa82d0bdc4" containerID="a85ecf9804f6d9baa0493f769d8f30c8027fa314822463e013f412297813d408" exitCode=0
Oct 31 13:14:02.458004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.425503    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp" event={"ID":"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4","Type":"ContainerDied","Data":"a85ecf9804f6d9baa0493f769d8f30c8027fa314822463e013f412297813d408"}
Oct 31 13:14:02.530733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.516655    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:14:02.530733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.516740    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:14:02.557733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.541179    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:02.580732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.568973    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:14:02.580732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.569035    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:14:02.597855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.588534    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:14:02.597855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.588593    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:14:02.629336 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.629293    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:14:02.653403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.653290    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:02.789785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.778464    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aea6021-b24xb\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:02.948855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.913728    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:14:02.948855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.913793    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:14:02.953723 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.951894    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:02.968758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.955048    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:02.968758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.955120    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:14:02.970820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:02.969139    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.027725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.027680    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvpxl\" (UniqueName: \"kubernetes.io/projected/641a74ef-ffdc-417d-86be-b38c42d67de3-kube-api-access-gvpxl\") pod \"maas-api-revoke-sub-3aea6021-b24xb\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:03.197818 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.185269    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.275108 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.185364    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.296819 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.185390    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.298030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.214241    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.298812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.298784    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:03.409734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.247641    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.268735    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.557668    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.538231    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"]
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.557873    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.214278    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.461506    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.559901    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.506822    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.559965    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.538281    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.568788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.562689    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.587732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.571556    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:14:03.628652 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.592842    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:14:03.660836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.660797    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:14:03.664805 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.653971    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.693731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.653677    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:14:03.752786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.733167    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:03.813731 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:03.812851    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841f919b_9011_4970_98aa_2d0cb8fe36f6.slice/crio-9c3d1322607cdcfece2a071a32361849cd65c64a040c21dc295dd1fe85abc970 WatchSource:0}: Error finding container 9c3d1322607cdcfece2a071a32361849cd65c64a040c21dc295dd1fe85abc970: Status 404 returned error can't find the container with id 9c3d1322607cdcfece2a071a32361849cd65c64a040c21dc295dd1fe85abc970
Oct 31 13:14:03.993781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.979853    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Oct 31 13:14:03.993781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.979920    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Oct 31 13:14:03.993781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:03.981151    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:04.029754 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.012155    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:04.125492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.125462    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:14:04.269963 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.269469    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:14:04.269963 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.269544    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:14:04.302259 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.296800    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:04.302259 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.298056    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:04.302259 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.298730    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:04.531088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.472733    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:14:04.531088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.472795    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:14:04.531088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.501147    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:14:04.531088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.516387    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:14:04.531088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.516452    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:14:04.535038 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.534964    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:14:04.623140 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.596598    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:04.623140 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.597788    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 31 13:14:04.623140 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.597833    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 31 13:14:04.623140 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.598235    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:04.650047 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.648835    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:14:04.666350 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.666305    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:04.848153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.848051    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:04.986026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:04.985972    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:05.000759 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.000141    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:14:05.054732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.053820    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:05.108789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.107787    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-xks-ca\") pod \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") "
Oct 31 13:14:05.108789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.107895    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-kube-api-access-7sssl\" (UniqueName: \"kubernetes.io/projected/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-kube-api-access-7sssl\") pod \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") "
Oct 31 13:14:05.108789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.108048    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-openshift-ca\") pod \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\" (UID: \"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4\") "
Oct 31 13:14:05.125736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.123574    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-xks-ca" pod "a496c77d-33bb-4c88-8de7-1ffa82d0bdc4" (UID: "a496c77d-33bb-4c88-8de7-1ffa82d0bdc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:14:05.126779 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.126651    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-kube-api-access-7sssl" pod "a496c77d-33bb-4c88-8de7-1ffa82d0bdc4" (UID: "a496c77d-33bb-4c88-8de7-1ffa82d0bdc4"). InnerVolumeSpecName "kube-api-access-7sssl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:14:05.127743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.127639    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-openshift-ca" pod "a496c77d-33bb-4c88-8de7-1ffa82d0bdc4" (UID: "a496c77d-33bb-4c88-8de7-1ffa82d0bdc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:14:05.211358 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.197740    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"
Oct 31 13:14:05.260661 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.260627    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 31 13:14:05.260956 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.260929    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 31 13:14:05.264268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.264246    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:05.280761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.272085    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:05.280761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.272115    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp" event={"ID":"a496c77d-33bb-4c88-8de7-1ffa82d0bdc4","Type":"ContainerDied","Data":"5471e4a5dc57e5aaedabfbce269a1bdf23a215c85bff07eb2f0f4c497bb5fca5"}
Oct 31 13:14:05.280761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.272155    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5471e4a5dc57e5aaedabfbce269a1bdf23a215c85bff07eb2f0f4c497bb5fca5"
Oct 31 13:14:05.280761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.272168    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k" event={"ID":"841f919b-9011-4970-98aa-2d0cb8fe36f6","Type":"ContainerStarted","Data":"9c3d1322607cdcfece2a071a32361849cd65c64a040c21dc295dd1fe85abc970"}
Oct 31 13:14:05.280761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.272215    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:14:05.280761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.272245    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:05.280761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.272273    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:14:05.280761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.272305    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:14:05.294290 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.294261    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sssl\" (UniqueName: \"kubernetes.io/projected/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-kube-api-access-7sssl\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:05.294521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.294504    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:05.294612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.294601    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:05.307231 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.307205    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:14:05.461198 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.461094    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"]
Oct 31 13:14:05.465789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.464936    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Oct 31 13:14:05.465789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.464988    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Oct 31 13:14:05.499008 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.494419    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 31 13:14:05.499008 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.494495    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 31 13:14:05.571451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:05.571381    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k" podStartSLOduration=5.5713601409999995 podStartE2EDuration="5.571360141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:05.548887133 +0000 UTC m=+824.964828953" watchObservedRunningTime="2026-10-31 13:14:05.571360141 +0000 UTC m=+824.987301982"
Oct 31 13:14:06.074296 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.065368    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:06.074296 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.065432    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:06.149742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.147163    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:14:06.149742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.148825    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:14:06.159739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.159060    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:14:06.297774 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.297740    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:14:06.298891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.298864    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:14:06.340984 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.339132    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:14:06.467503 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.467442    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb" event={"ID":"641a74ef-ffdc-417d-86be-b38c42d67de3","Type":"ContainerStarted","Data":"6f984db389bba53d67958e1ffb1dc998198b12a6788d6c9ec5433bfd95294bc6"}
Oct 31 13:14:06.587891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.587842    5787 generic.go:356] "Generic (PLEG): container finished" podID="841f919b-9011-4970-98aa-2d0cb8fe36f6" containerID="3640241891b9c4525a965ee8e0fc579d6a4a55ef6c4f8e1ed1b9da4e19bf4b5f" exitCode=0
Oct 31 13:14:06.588675 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.588213    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k" event={"ID":"841f919b-9011-4970-98aa-2d0cb8fe36f6","Type":"ContainerDied","Data":"3640241891b9c4525a965ee8e0fc579d6a4a55ef6c4f8e1ed1b9da4e19bf4b5f"}
Oct 31 13:14:06.722363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.721985    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:14:06.726914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.723439    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:14:06.747937 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.747172    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:14:06.822902 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.822072    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:14:06.823658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.823633    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:14:06.830472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:06.830443    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:14:07.058189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.058098    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:14:07.058630 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.058612    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:14:07.059459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.059430    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:14:07.087339 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.087296    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"]
Oct 31 13:14:07.088159 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.088135    5787 cpu_manager.go:414] "containerMap: removing container" podUID="a496c77d-33bb-4c88-8de7-1ffa82d0bdc4" containerName="revoke-keys"
Oct 31 13:14:07.088159 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.088159    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a496c77d-33bb-4c88-8de7-1ffa82d0bdc4" containerName="revoke-keys"
Oct 31 13:14:07.088396 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.088338    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="a496c77d-33bb-4c88-8de7-1ffa82d0bdc4" containerName="revoke-keys"
Oct 31 13:14:07.092949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.092926    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.107744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.107674    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"]
Oct 31 13:14:07.213641 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.213598    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.213883 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.213836    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.213945 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.213918    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.214026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.213993    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.214085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.214049    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sjj4\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-kube-api-access-6sjj4\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.214230 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.214201    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.214367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.214273    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.214367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.214302    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.214367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.214335    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.316694    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.316882    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.316927    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.316993    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.317032    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sjj4\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-kube-api-access-6sjj4\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.317095    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.317150    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.317181    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.317210    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.317697    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-certs\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.331787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.317972    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-data\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.337410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.333333    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-credential-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.337410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.333591    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-socket\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.340083 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.339053    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-envoy\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.340083 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.340002    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istiod-ca-cert\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.341649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.341626    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-podinfo\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.371824 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.371794    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sjj4\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-kube-api-access-6sjj4\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.374268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.374237    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-token\") pod \"e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") " pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.425921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.425882    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:07.598468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.598381    5787 generic.go:356] "Generic (PLEG): container finished" podID="641a74ef-ffdc-417d-86be-b38c42d67de3" containerID="0cb70b12161ff86d85b587f2dafd5a3a957c6fc4be806f84a0e9fb0fdd56e143" exitCode=0
Oct 31 13:14:07.598619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.598466    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb" event={"ID":"641a74ef-ffdc-417d-86be-b38c42d67de3","Type":"ContainerDied","Data":"0cb70b12161ff86d85b587f2dafd5a3a957c6fc4be806f84a0e9fb0fdd56e143"}
Oct 31 13:14:07.718214 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:07.718187    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e8209df_eac1_4874_a784_fe7e9a33a7fa.slice/crio-6f5ab0540ecba6a64e2db307dc809089f1e00cdbbbe902d0ac934ef1c7369c42 WatchSource:0}: Error finding container 6f5ab0540ecba6a64e2db307dc809089f1e00cdbbbe902d0ac934ef1c7369c42: Status 404 returned error can't find the container with id 6f5ab0540ecba6a64e2db307dc809089f1e00cdbbbe902d0ac934ef1c7369c42
Oct 31 13:14:07.719460 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.719431    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"]
Oct 31 13:14:07.720874 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.720836    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:07.720943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.720907    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:07.720943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.720936    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:07.753405 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.753381    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:07.829056 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.829014    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/841f919b-9011-4970-98aa-2d0cb8fe36f6-kube-api-access-r8llm\" (UniqueName: \"kubernetes.io/projected/841f919b-9011-4970-98aa-2d0cb8fe36f6-kube-api-access-r8llm\") pod \"841f919b-9011-4970-98aa-2d0cb8fe36f6\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") "
Oct 31 13:14:07.829258 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.829100    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-xks-ca\") pod \"841f919b-9011-4970-98aa-2d0cb8fe36f6\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") "
Oct 31 13:14:07.829258 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.829155    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-openshift-ca\") pod \"841f919b-9011-4970-98aa-2d0cb8fe36f6\" (UID: \"841f919b-9011-4970-98aa-2d0cb8fe36f6\") "
Oct 31 13:14:07.830670 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.830011    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-openshift-ca" pod "841f919b-9011-4970-98aa-2d0cb8fe36f6" (UID: "841f919b-9011-4970-98aa-2d0cb8fe36f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:14:07.831980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.831811    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841f919b-9011-4970-98aa-2d0cb8fe36f6-kube-api-access-r8llm" pod "841f919b-9011-4970-98aa-2d0cb8fe36f6" (UID: "841f919b-9011-4970-98aa-2d0cb8fe36f6"). InnerVolumeSpecName "kube-api-access-r8llm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:14:07.832262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.832242    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-xks-ca" pod "841f919b-9011-4970-98aa-2d0cb8fe36f6" (UID: "841f919b-9011-4970-98aa-2d0cb8fe36f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:14:07.931481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.931403    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8llm\" (UniqueName: \"kubernetes.io/projected/841f919b-9011-4970-98aa-2d0cb8fe36f6-kube-api-access-r8llm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:07.931481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.931434    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:07.931481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:07.931444    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/841f919b-9011-4970-98aa-2d0cb8fe36f6-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:08.624511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.624455    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"
Oct 31 13:14:08.624511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.624501    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k" event={"ID":"841f919b-9011-4970-98aa-2d0cb8fe36f6","Type":"ContainerDied","Data":"9c3d1322607cdcfece2a071a32361849cd65c64a040c21dc295dd1fe85abc970"}
Oct 31 13:14:08.625080 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.624553    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c3d1322607cdcfece2a071a32361849cd65c64a040c21dc295dd1fe85abc970"
Oct 31 13:14:08.626743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.626674    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" event={"ID":"0e8209df-eac1-4874-a784-fe7e9a33a7fa","Type":"ContainerStarted","Data":"a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83"}
Oct 31 13:14:08.626743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.626744    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" event={"ID":"0e8209df-eac1-4874-a784-fe7e9a33a7fa","Type":"ContainerStarted","Data":"6f5ab0540ecba6a64e2db307dc809089f1e00cdbbbe902d0ac934ef1c7369c42"}
Oct 31 13:14:08.681865 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.681804    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podStartSLOduration=1.681787105 podStartE2EDuration="1.681787105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:08.681434003 +0000 UTC m=+828.097375836" watchObservedRunningTime="2026-10-31 13:14:08.681787105 +0000 UTC m=+828.097728941"
Oct 31 13:14:08.824835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.824803    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:08.964207 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.964176    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641a74ef-ffdc-417d-86be-b38c42d67de3-kube-api-access-gvpxl\" (UniqueName: \"kubernetes.io/projected/641a74ef-ffdc-417d-86be-b38c42d67de3-kube-api-access-gvpxl\") pod \"641a74ef-ffdc-417d-86be-b38c42d67de3\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") "
Oct 31 13:14:08.964423 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.964358    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-xks-ca\") pod \"641a74ef-ffdc-417d-86be-b38c42d67de3\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") "
Oct 31 13:14:08.964502 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.964467    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-openshift-ca\") pod \"641a74ef-ffdc-417d-86be-b38c42d67de3\" (UID: \"641a74ef-ffdc-417d-86be-b38c42d67de3\") "
Oct 31 13:14:08.964903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.964878    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-openshift-ca" pod "641a74ef-ffdc-417d-86be-b38c42d67de3" (UID: "641a74ef-ffdc-417d-86be-b38c42d67de3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:14:08.966495 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.966469    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641a74ef-ffdc-417d-86be-b38c42d67de3-kube-api-access-gvpxl" pod "641a74ef-ffdc-417d-86be-b38c42d67de3" (UID: "641a74ef-ffdc-417d-86be-b38c42d67de3"). InnerVolumeSpecName "kube-api-access-gvpxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:14:08.966922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:08.966903    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-xks-ca" pod "641a74ef-ffdc-417d-86be-b38c42d67de3" (UID: "641a74ef-ffdc-417d-86be-b38c42d67de3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:14:09.066491 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:09.066449    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:09.066491 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:09.066490    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvpxl\" (UniqueName: \"kubernetes.io/projected/641a74ef-ffdc-417d-86be-b38c42d67de3-kube-api-access-gvpxl\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:09.066739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:09.066506    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/641a74ef-ffdc-417d-86be-b38c42d67de3-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:09.426586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:09.426487    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:09.427943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:09.427915    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:09.428081 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:09.427974    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:09.661110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:09.661071    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb" event={"ID":"641a74ef-ffdc-417d-86be-b38c42d67de3","Type":"ContainerDied","Data":"6f984db389bba53d67958e1ffb1dc998198b12a6788d6c9ec5433bfd95294bc6"}
Oct 31 13:14:09.661653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:09.661633    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f984db389bba53d67958e1ffb1dc998198b12a6788d6c9ec5433bfd95294bc6"
Oct 31 13:14:09.661822 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:09.661088    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"
Oct 31 13:14:10.426525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:10.426484    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:10.426745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:10.426546    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:11.430188 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:11.430148    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:11.430914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:11.430875    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:12.242734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:12.241841    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:12.246742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:12.246659    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:12.479741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:12.477436    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:12.494793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:12.491493    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:12.494793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:12.491604    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:12.656730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:12.650901    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:12.825471 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:12.825034    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:12.943895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:12.943844    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.243455 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.243396    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.244580 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.244557    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:14:13.254198 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.254159    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.255694 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.254781    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:14:13.401805 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.384878    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.430607 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.430147    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:13.430607 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.430211    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:13.519989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.486809    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.519989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.488594    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:14:13.565734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.544540    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.666731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.662868    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.691735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.688821    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:13.759734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.756300    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.889790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.875442    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.889790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.879738    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 31 13:14:13.889790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.881859    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:14:13.974744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.965975    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:13.974744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.968211    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:14:13.978755 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:13.976918    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:14:14.068737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.067855    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:14.125370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.112883    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:14.125370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.112953    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:14.310046 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.267104    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:14.310046 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.267490    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:14.470927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.425849    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:14.470927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.434843    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:14:14.470927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.440247    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:14.470927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.440300    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:14.497747 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.490581    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:14.551756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.545272    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:14.572847 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.570791    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Oct 31 13:14:14.663871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.663819    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:14.977421 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.977360    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:14.978074 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:14.978043    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.012085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.012043    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.035862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.035796    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.078167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.078116    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.119226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.119190    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:15.119546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.119529    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:14:15.119674 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.119277    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.121369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.121348    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:14:15.137082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.137028    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.334789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.334215    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.351734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.348530    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.351734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.349232    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.430215 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.430181    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:15.430529 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.430486    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:15.550484 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.550421    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.565142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.564508    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:14:15.579440 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.579390    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:15.627999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.627892    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.890748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.890608    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.896255 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.896199    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.965927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.965876    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:15.966244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.966196    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.995271 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.995208    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:15.996026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:15.995979    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:14:16.015318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.012775    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.016526 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.016483    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:16.016783 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.016744    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.117753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.094814    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.117753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.095117    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.124732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.122785    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:16.124732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.123653    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:14:16.334898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.334851    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.363738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.355242    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:16.401458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.401420    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:16.401718 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.401673    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.427105 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.427069    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:16.427402 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.427371    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:16.447733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.445808    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:16.447733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.445876    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.553134 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.553087    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.597000 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.596907    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:16.871099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.870988    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.895733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.895460    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:14:16.904742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.896790    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:16.904742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:16.897168    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:14:17.004232 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.004184    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 31 13:14:17.064798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.047011    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Oct 31 13:14:17.113650 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.054640    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:17.167107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.120829    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:17.167855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.120879    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:17.168267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.163823    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:17.168423 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.168395    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:17.168540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.163887    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:17.168945 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.168923    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:14:17.398734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.383134    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:17.427594 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.399372    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:17.456838 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.456793    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:17.457094 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.452151    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Oct 31 13:14:17.457206 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.453178    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:17.457584 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.413628    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:17.457804 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.457788    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:14:17.462784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.440369    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:17.463737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.463683    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:17.561436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.561397    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:17.569744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.565963    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:17.569744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.566018    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:17.684932 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.678825    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:17.917119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.917075    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:17.923105 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:17.917824    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:14:18.044372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.044330    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:18.087738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.052828    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": context deadline exceeded"
Oct 31 13:14:18.187852 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.187804    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:18.198732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.188986    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:18.198732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.189294    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:18.278738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.278332    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:18.446788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.426914    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:18.509731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.471219    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:18.509731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.471804    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:18.509731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.471851    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:18.509731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.472272    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:18.558949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.517809    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"]
Oct 31 13:14:18.558949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.518666    5787 cpu_manager.go:414] "containerMap: removing container" podUID="641a74ef-ffdc-417d-86be-b38c42d67de3" containerName="revoke-keys"
Oct 31 13:14:18.558949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.518687    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641a74ef-ffdc-417d-86be-b38c42d67de3" containerName="revoke-keys"
Oct 31 13:14:18.558949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.518759    5787 cpu_manager.go:414] "containerMap: removing container" podUID="841f919b-9011-4970-98aa-2d0cb8fe36f6" containerName="revoke-keys"
Oct 31 13:14:18.558949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.518771    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="841f919b-9011-4970-98aa-2d0cb8fe36f6" containerName="revoke-keys"
Oct 31 13:14:18.558949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.518986    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="841f919b-9011-4970-98aa-2d0cb8fe36f6" containerName="revoke-keys"
Oct 31 13:14:18.558949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.519005    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="641a74ef-ffdc-417d-86be-b38c42d67de3" containerName="revoke-keys"
Oct 31 13:14:18.582733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.563005    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:14:18.582733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.563073    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:14:18.582733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.564007    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:18.618730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.597440    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"]
Oct 31 13:14:18.619029 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:14:18.597526    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:14:18.619185 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.610555    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:14:18.619299 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.610779    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:14:18.619401 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.610886    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:14:18.619480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.610473    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:14:18.696744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.681527    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:18.726546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.726450    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7428758a-m4k4g\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:18.726851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.726828    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7428758a-m4k4g\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:18.727007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.726993    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjhtd\" (UniqueName: \"kubernetes.io/projected/0b7797c9-9794-4086-a59f-be440d6c72ad-kube-api-access-xjhtd\") pod \"maas-api-revoke-sub-7428758a-m4k4g\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:18.964163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.902899    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7428758a-m4k4g\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:18.964163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.903001    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7428758a-m4k4g\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:18.964163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.903069    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjhtd\" (UniqueName: \"kubernetes.io/projected/0b7797c9-9794-4086-a59f-be440d6c72ad-kube-api-access-xjhtd\") pod \"maas-api-revoke-sub-7428758a-m4k4g\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:18.964163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.924466    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:18.964163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.961844    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:18.993217 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:18.993141    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7428758a-m4k4g\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:19.009613 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.009583    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:14:19.015716 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.015669    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:14:19.022631 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.022600    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:14:19.022795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.022725    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:14:19.029254 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.029228    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:14:19.088084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.087315    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjhtd\" (UniqueName: \"kubernetes.io/projected/0b7797c9-9794-4086-a59f-be440d6c72ad-kube-api-access-xjhtd\") pod \"maas-api-revoke-sub-7428758a-m4k4g\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:19.113810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.110015    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:19.161843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.158407    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:14:19.181189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.181161    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7428758a-m4k4g\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:19.214368 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.214326    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:19.287124 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.285398    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:14:19.291235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.290212    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:14:19.299670 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.299363    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:19.430028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.429175    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Oct 31 13:14:19.430028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.429238    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Oct 31 13:14:19.627144 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:19.627116    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"]
Oct 31 13:14:19.628289 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:19.628263    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7797c9_9794_4086_a59f_be440d6c72ad.slice/crio-391e24c2dc3775871754a7674f3a1fb4a07b4cfbe186387d13100250dfd77d50 WatchSource:0}: Error finding container 391e24c2dc3775871754a7674f3a1fb4a07b4cfbe186387d13100250dfd77d50: Status 404 returned error can't find the container with id 391e24c2dc3775871754a7674f3a1fb4a07b4cfbe186387d13100250dfd77d50
Oct 31 13:14:20.109931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:20.109682    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g" event={"ID":"0b7797c9-9794-4086-a59f-be440d6c72ad","Type":"ContainerStarted","Data":"969dc7621a11288b6903fe19bdefdd03198126b299026d53397759bae36bf102"}
Oct 31 13:14:20.109931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:20.109772    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g" event={"ID":"0b7797c9-9794-4086-a59f-be440d6c72ad","Type":"ContainerStarted","Data":"391e24c2dc3775871754a7674f3a1fb4a07b4cfbe186387d13100250dfd77d50"}
Oct 31 13:14:20.132114 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:20.132051    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g" podStartSLOduration=2.132028829 podStartE2EDuration="2.132028829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:20.131658755 +0000 UTC m=+839.547600596" watchObservedRunningTime="2026-10-31 13:14:20.132028829 +0000 UTC m=+839.547970669"
Oct 31 13:14:20.553174 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:20.553145    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:20.553669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:20.553649    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:20.555201 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:20.555167    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:14:21.120058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:21.120017    5787 generic.go:356] "Generic (PLEG): container finished" podID="0b7797c9-9794-4086-a59f-be440d6c72ad" containerID="969dc7621a11288b6903fe19bdefdd03198126b299026d53397759bae36bf102" exitCode=0
Oct 31 13:14:21.120468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:21.120069    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g" event={"ID":"0b7797c9-9794-4086-a59f-be440d6c72ad","Type":"ContainerDied","Data":"969dc7621a11288b6903fe19bdefdd03198126b299026d53397759bae36bf102"}
Oct 31 13:14:22.393241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.393214    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:22.514371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.514341    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7797c9-9794-4086-a59f-be440d6c72ad-kube-api-access-xjhtd\" (UniqueName: \"kubernetes.io/projected/0b7797c9-9794-4086-a59f-be440d6c72ad-kube-api-access-xjhtd\") pod \"0b7797c9-9794-4086-a59f-be440d6c72ad\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") "
Oct 31 13:14:22.514681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.514381    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-openshift-ca\") pod \"0b7797c9-9794-4086-a59f-be440d6c72ad\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") "
Oct 31 13:14:22.514681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.514415    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-xks-ca\") pod \"0b7797c9-9794-4086-a59f-be440d6c72ad\" (UID: \"0b7797c9-9794-4086-a59f-be440d6c72ad\") "
Oct 31 13:14:22.514960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.514814    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-openshift-ca" pod "0b7797c9-9794-4086-a59f-be440d6c72ad" (UID: "0b7797c9-9794-4086-a59f-be440d6c72ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:14:22.515229 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.515169    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:22.516693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.516666    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-xks-ca" pod "0b7797c9-9794-4086-a59f-be440d6c72ad" (UID: "0b7797c9-9794-4086-a59f-be440d6c72ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:14:22.516851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.516836    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7797c9-9794-4086-a59f-be440d6c72ad-kube-api-access-xjhtd" pod "0b7797c9-9794-4086-a59f-be440d6c72ad" (UID: "0b7797c9-9794-4086-a59f-be440d6c72ad"). InnerVolumeSpecName "kube-api-access-xjhtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:14:22.616999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.616967    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjhtd\" (UniqueName: \"kubernetes.io/projected/0b7797c9-9794-4086-a59f-be440d6c72ad-kube-api-access-xjhtd\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:22.616999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:22.616997    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b7797c9-9794-4086-a59f-be440d6c72ad-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:23.186617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.186587    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"
Oct 31 13:14:23.186617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.186609    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g" event={"ID":"0b7797c9-9794-4086-a59f-be440d6c72ad","Type":"ContainerDied","Data":"391e24c2dc3775871754a7674f3a1fb4a07b4cfbe186387d13100250dfd77d50"}
Oct 31 13:14:23.186853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.186658    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="391e24c2dc3775871754a7674f3a1fb4a07b4cfbe186387d13100250dfd77d50"
Oct 31 13:14:23.952851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.952816    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"]
Oct 31 13:14:23.953394 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.953382    5787 cpu_manager.go:414] "containerMap: removing container" podUID="0b7797c9-9794-4086-a59f-be440d6c72ad" containerName="revoke-keys"
Oct 31 13:14:23.953442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.953396    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7797c9-9794-4086-a59f-be440d6c72ad" containerName="revoke-keys"
Oct 31 13:14:23.953518 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.953508    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7797c9-9794-4086-a59f-be440d6c72ad" containerName="revoke-keys"
Oct 31 13:14:23.957439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.957419    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:23.963247 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.963210    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:14:23.963387 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.963340    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:14:23.963447 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.963389    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:14:23.963523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.963507    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:14:23.967898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:23.967875    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"]
Oct 31 13:14:24.038897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.038865    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72vgm\" (UniqueName: \"kubernetes.io/projected/f07906e9-d37b-4cad-8296-c9104942cdbc-kube-api-access-72vgm\") pod \"maas-api-revoke-sub-46cc47a8-pgzh9\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.039070 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.038909    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46cc47a8-pgzh9\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.039115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.039092    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46cc47a8-pgzh9\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.140417 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.140379    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46cc47a8-pgzh9\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.140597 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.140514    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72vgm\" (UniqueName: \"kubernetes.io/projected/f07906e9-d37b-4cad-8296-c9104942cdbc-kube-api-access-72vgm\") pod \"maas-api-revoke-sub-46cc47a8-pgzh9\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.140597 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.140545    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46cc47a8-pgzh9\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.141258 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.141230    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46cc47a8-pgzh9\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.143100 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.143082    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46cc47a8-pgzh9\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.153319 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.153283    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72vgm\" (UniqueName: \"kubernetes.io/projected/f07906e9-d37b-4cad-8296-c9104942cdbc-kube-api-access-72vgm\") pod \"maas-api-revoke-sub-46cc47a8-pgzh9\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.273672 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.273590    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:24.571295 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.571264    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"]
Oct 31 13:14:24.572081 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:24.572022    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07906e9_d37b_4cad_8296_c9104942cdbc.slice/crio-aea469408f343cade946cca868376f0a41162e0ca68d9dd4532c8c08d750cbb2 WatchSource:0}: Error finding container aea469408f343cade946cca868376f0a41162e0ca68d9dd4532c8c08d750cbb2: Status 404 returned error can't find the container with id aea469408f343cade946cca868376f0a41162e0ca68d9dd4532c8c08d750cbb2
Oct 31 13:14:24.764798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.764765    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"]
Oct 31 13:14:24.770172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.770150    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:24.778999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.778967    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"]
Oct 31 13:14:24.855303 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.855268    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zlsh\" (UniqueName: \"kubernetes.io/projected/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-kube-api-access-2zlsh\") pod \"maas-api-revoke-sub-64dfbcc8-v9t2c\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:24.855497 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.855327    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64dfbcc8-v9t2c\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:24.855497 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.855479    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64dfbcc8-v9t2c\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:24.956414 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.956382    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zlsh\" (UniqueName: \"kubernetes.io/projected/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-kube-api-access-2zlsh\") pod \"maas-api-revoke-sub-64dfbcc8-v9t2c\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:24.956929 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.956439    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64dfbcc8-v9t2c\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:24.956929 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.956536    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64dfbcc8-v9t2c\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:24.957157 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.957136    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64dfbcc8-v9t2c\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:24.959211 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.959184    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64dfbcc8-v9t2c\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:24.971378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:24.971343    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zlsh\" (UniqueName: \"kubernetes.io/projected/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-kube-api-access-2zlsh\") pod \"maas-api-revoke-sub-64dfbcc8-v9t2c\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:25.087351 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:25.087313    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:25.208766 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:25.208724    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9" event={"ID":"f07906e9-d37b-4cad-8296-c9104942cdbc","Type":"ContainerStarted","Data":"506c96ca7c9f4a6cf4040ebd241d7af03566694d2c8d28e8d7f293c03f2a4729"}
Oct 31 13:14:25.208979 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:25.208810    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9" event={"ID":"f07906e9-d37b-4cad-8296-c9104942cdbc","Type":"ContainerStarted","Data":"aea469408f343cade946cca868376f0a41162e0ca68d9dd4532c8c08d750cbb2"}
Oct 31 13:14:25.282859 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:25.282322    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9" podStartSLOduration=2.2822972999999998 podStartE2EDuration="2.2822973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:25.232225013 +0000 UTC m=+844.648166853" watchObservedRunningTime="2026-10-31 13:14:25.2822973 +0000 UTC m=+844.698239141"
Oct 31 13:14:25.379834 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:25.379799    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7253c9_9e35_449b_ac5a_73b9bfc583fa.slice/crio-a46c52a3d1d3070e8a3378c006086d1d23bfc0f0021bf4e32b7af248d48180d6 WatchSource:0}: Error finding container a46c52a3d1d3070e8a3378c006086d1d23bfc0f0021bf4e32b7af248d48180d6: Status 404 returned error can't find the container with id a46c52a3d1d3070e8a3378c006086d1d23bfc0f0021bf4e32b7af248d48180d6
Oct 31 13:14:25.380019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:25.379995    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"]
Oct 31 13:14:26.218821 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:26.218784    5787 generic.go:356] "Generic (PLEG): container finished" podID="8d7253c9-9e35-449b-ac5a-73b9bfc583fa" containerID="d70b77e1fc84307ee70b0805ed4cb8fdf348cf15ec2813297454e26b8607e5d5" exitCode=0
Oct 31 13:14:26.219272 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:26.218832    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c" event={"ID":"8d7253c9-9e35-449b-ac5a-73b9bfc583fa","Type":"ContainerDied","Data":"d70b77e1fc84307ee70b0805ed4cb8fdf348cf15ec2813297454e26b8607e5d5"}
Oct 31 13:14:26.219272 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:26.218892    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c" event={"ID":"8d7253c9-9e35-449b-ac5a-73b9bfc583fa","Type":"ContainerStarted","Data":"a46c52a3d1d3070e8a3378c006086d1d23bfc0f0021bf4e32b7af248d48180d6"}
Oct 31 13:14:26.220821 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:26.220795    5787 generic.go:356] "Generic (PLEG): container finished" podID="f07906e9-d37b-4cad-8296-c9104942cdbc" containerID="506c96ca7c9f4a6cf4040ebd241d7af03566694d2c8d28e8d7f293c03f2a4729" exitCode=0
Oct 31 13:14:26.220974 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:26.220841    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9" event={"ID":"f07906e9-d37b-4cad-8296-c9104942cdbc","Type":"ContainerDied","Data":"506c96ca7c9f4a6cf4040ebd241d7af03566694d2c8d28e8d7f293c03f2a4729"}
Oct 31 13:14:27.524165 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.524139    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:27.527585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.527547    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:27.586408 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.586364    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-openshift-ca\") pod \"f07906e9-d37b-4cad-8296-c9104942cdbc\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") "
Oct 31 13:14:27.586609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.586420    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-openshift-ca\") pod \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") "
Oct 31 13:14:27.586609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.586446    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-xks-ca\") pod \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") "
Oct 31 13:14:27.586609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.586521    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-xks-ca\") pod \"f07906e9-d37b-4cad-8296-c9104942cdbc\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") "
Oct 31 13:14:27.586609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.586562    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f07906e9-d37b-4cad-8296-c9104942cdbc-kube-api-access-72vgm\" (UniqueName: \"kubernetes.io/projected/f07906e9-d37b-4cad-8296-c9104942cdbc-kube-api-access-72vgm\") pod \"f07906e9-d37b-4cad-8296-c9104942cdbc\" (UID: \"f07906e9-d37b-4cad-8296-c9104942cdbc\") "
Oct 31 13:14:27.586897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.586635    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-kube-api-access-2zlsh\" (UniqueName: \"kubernetes.io/projected/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-kube-api-access-2zlsh\") pod \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\" (UID: \"8d7253c9-9e35-449b-ac5a-73b9bfc583fa\") "
Oct 31 13:14:27.586897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.586857    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-openshift-ca" pod "f07906e9-d37b-4cad-8296-c9104942cdbc" (UID: "f07906e9-d37b-4cad-8296-c9104942cdbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:14:27.586897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.586879    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-openshift-ca" pod "8d7253c9-9e35-449b-ac5a-73b9bfc583fa" (UID: "8d7253c9-9e35-449b-ac5a-73b9bfc583fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:14:27.587407 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.587306    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:27.587407 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.587335    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:27.589161 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.589125    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-xks-ca" pod "f07906e9-d37b-4cad-8296-c9104942cdbc" (UID: "f07906e9-d37b-4cad-8296-c9104942cdbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:14:27.589416 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.589392    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-kube-api-access-2zlsh" pod "8d7253c9-9e35-449b-ac5a-73b9bfc583fa" (UID: "8d7253c9-9e35-449b-ac5a-73b9bfc583fa"). InnerVolumeSpecName "kube-api-access-2zlsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:14:27.590113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.590088    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-xks-ca" pod "8d7253c9-9e35-449b-ac5a-73b9bfc583fa" (UID: "8d7253c9-9e35-449b-ac5a-73b9bfc583fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:14:27.590223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.590131    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f07906e9-d37b-4cad-8296-c9104942cdbc-kube-api-access-72vgm" pod "f07906e9-d37b-4cad-8296-c9104942cdbc" (UID: "f07906e9-d37b-4cad-8296-c9104942cdbc"). InnerVolumeSpecName "kube-api-access-72vgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:14:27.688521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.688486    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72vgm\" (UniqueName: \"kubernetes.io/projected/f07906e9-d37b-4cad-8296-c9104942cdbc-kube-api-access-72vgm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:27.688521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.688522    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zlsh\" (UniqueName: \"kubernetes.io/projected/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-kube-api-access-2zlsh\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:27.688735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.688538    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d7253c9-9e35-449b-ac5a-73b9bfc583fa-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:27.688735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:27.688552    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f07906e9-d37b-4cad-8296-c9104942cdbc-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:28.258899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:28.257396    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9" event={"ID":"f07906e9-d37b-4cad-8296-c9104942cdbc","Type":"ContainerDied","Data":"aea469408f343cade946cca868376f0a41162e0ca68d9dd4532c8c08d750cbb2"}
Oct 31 13:14:28.258899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:28.257467    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aea469408f343cade946cca868376f0a41162e0ca68d9dd4532c8c08d750cbb2"
Oct 31 13:14:28.258899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:28.257554    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"
Oct 31 13:14:28.265878 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:28.265831    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c" event={"ID":"8d7253c9-9e35-449b-ac5a-73b9bfc583fa","Type":"ContainerDied","Data":"a46c52a3d1d3070e8a3378c006086d1d23bfc0f0021bf4e32b7af248d48180d6"}
Oct 31 13:14:28.266043 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:28.265895    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a46c52a3d1d3070e8a3378c006086d1d23bfc0f0021bf4e32b7af248d48180d6"
Oct 31 13:14:28.266043 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:28.265965    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"
Oct 31 13:14:30.083293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.081834    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"]
Oct 31 13:14:30.086399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.084042    5787 cpu_manager.go:414] "containerMap: removing container" podUID="8d7253c9-9e35-449b-ac5a-73b9bfc583fa" containerName="revoke-keys"
Oct 31 13:14:30.086399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.084069    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d7253c9-9e35-449b-ac5a-73b9bfc583fa" containerName="revoke-keys"
Oct 31 13:14:30.086399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.084160    5787 cpu_manager.go:414] "containerMap: removing container" podUID="f07906e9-d37b-4cad-8296-c9104942cdbc" containerName="revoke-keys"
Oct 31 13:14:30.086399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.084167    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f07906e9-d37b-4cad-8296-c9104942cdbc" containerName="revoke-keys"
Oct 31 13:14:30.086399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.084313    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="f07906e9-d37b-4cad-8296-c9104942cdbc" containerName="revoke-keys"
Oct 31 13:14:30.086399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.084326    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d7253c9-9e35-449b-ac5a-73b9bfc583fa" containerName="revoke-keys"
Oct 31 13:14:30.095501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.095463    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"]
Oct 31 13:14:30.095661 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.095589    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.106662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.106631    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:14:30.106662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.106656    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:14:30.106928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.106731    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:14:30.106928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.106777    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:14:30.142199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.142165    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28a55ca0-kms62\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.142403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.142321    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28a55ca0-kms62\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.142501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.142486    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d22v8\" (UniqueName: \"kubernetes.io/projected/744e9587-802d-495d-b30e-cec079e3f3f8-kube-api-access-d22v8\") pod \"maas-api-revoke-sub-28a55ca0-kms62\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.244228 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.244194    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28a55ca0-kms62\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.244432 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.244321    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d22v8\" (UniqueName: \"kubernetes.io/projected/744e9587-802d-495d-b30e-cec079e3f3f8-kube-api-access-d22v8\") pod \"maas-api-revoke-sub-28a55ca0-kms62\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.244432 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.244405    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28a55ca0-kms62\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.245114 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.245093    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28a55ca0-kms62\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.246741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.246719    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28a55ca0-kms62\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.258063 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.258040    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d22v8\" (UniqueName: \"kubernetes.io/projected/744e9587-802d-495d-b30e-cec079e3f3f8-kube-api-access-d22v8\") pod \"maas-api-revoke-sub-28a55ca0-kms62\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.417850 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.417760    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:30.710863 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:30.710834    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod744e9587_802d_495d_b30e_cec079e3f3f8.slice/crio-ad3f8633e7d3f9e5d8d7b5192388f4dd9f15a4f12d8f48580d6c6dbdc1c230c9 WatchSource:0}: Error finding container ad3f8633e7d3f9e5d8d7b5192388f4dd9f15a4f12d8f48580d6c6dbdc1c230c9: Status 404 returned error can't find the container with id ad3f8633e7d3f9e5d8d7b5192388f4dd9f15a4f12d8f48580d6c6dbdc1c230c9
Oct 31 13:14:30.711000 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:30.710937    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"]
Oct 31 13:14:31.291796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.291746    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62" event={"ID":"744e9587-802d-495d-b30e-cec079e3f3f8","Type":"ContainerStarted","Data":"2a1cf4987942614247b1a2f89cb9d6f19398db2cf1aef6cf8dda9265059b77fe"}
Oct 31 13:14:31.292198 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.291845    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62" event={"ID":"744e9587-802d-495d-b30e-cec079e3f3f8","Type":"ContainerStarted","Data":"ad3f8633e7d3f9e5d8d7b5192388f4dd9f15a4f12d8f48580d6c6dbdc1c230c9"}
Oct 31 13:14:31.315414 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.315360    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62" podStartSLOduration=1.31534624 podStartE2EDuration="1.31534624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:31.311844189 +0000 UTC m=+850.727786030" watchObservedRunningTime="2026-10-31 13:14:31.31534624 +0000 UTC m=+850.731288079"
Oct 31 13:14:31.568291 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.568257    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"]
Oct 31 13:14:31.573647 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.573620    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.590783 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.586588    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"]
Oct 31 13:14:31.590783 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.587108    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-426787\"/\"distinct-2-w4-426787-kserve-self-signed-certs\""
Oct 31 13:14:31.667753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.665561    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-home\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.667753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.665637    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-dshm\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.667753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.665799    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-tmp-dir\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.667753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.665840    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-model-cache\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.667753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.665996    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144afe47-505e-436b-a07d-5e1ea36818a1-tls-certs\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.667753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.666163    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5tq8\" (UniqueName: \"kubernetes.io/projected/144afe47-505e-436b-a07d-5e1ea36818a1-kube-api-access-s5tq8\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.767653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.767615    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-home\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.767836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.767689    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-dshm\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.767836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.767812    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-tmp-dir\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.767911 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.767840    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-model-cache\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.767945 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.767911    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144afe47-505e-436b-a07d-5e1ea36818a1-tls-certs\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.768086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.768067    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5tq8\" (UniqueName: \"kubernetes.io/projected/144afe47-505e-436b-a07d-5e1ea36818a1-kube-api-access-s5tq8\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.769183 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.769152    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-home\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.769183 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.769175    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-model-cache\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.769374 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.769349    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-tmp-dir\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.771251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.771228    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-dshm\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.771665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.771642    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144afe47-505e-436b-a07d-5e1ea36818a1-tls-certs\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.782889 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.782844    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5tq8\" (UniqueName: \"kubernetes.io/projected/144afe47-505e-436b-a07d-5e1ea36818a1-kube-api-access-s5tq8\") pod \"distinct-2-w4-426787-kserve-849b5994b-fpqgk\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") " pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:31.917309 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:31.916020    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:32.301615 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:32.301581    5787 generic.go:356] "Generic (PLEG): container finished" podID="744e9587-802d-495d-b30e-cec079e3f3f8" containerID="2a1cf4987942614247b1a2f89cb9d6f19398db2cf1aef6cf8dda9265059b77fe" exitCode=0
Oct 31 13:14:32.302029 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:32.301665    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62" event={"ID":"744e9587-802d-495d-b30e-cec079e3f3f8","Type":"ContainerDied","Data":"2a1cf4987942614247b1a2f89cb9d6f19398db2cf1aef6cf8dda9265059b77fe"}
Oct 31 13:14:32.361994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:32.361965    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"]
Oct 31 13:14:32.362700 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:32.362666    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod144afe47_505e_436b_a07d_5e1ea36818a1.slice/crio-769e0f85c9adac4b22f2538a72747c4e700e733dbef0cac3d9d3cf14a99cc4b4 WatchSource:0}: Error finding container 769e0f85c9adac4b22f2538a72747c4e700e733dbef0cac3d9d3cf14a99cc4b4: Status 404 returned error can't find the container with id 769e0f85c9adac4b22f2538a72747c4e700e733dbef0cac3d9d3cf14a99cc4b4
Oct 31 13:14:33.311491 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.311453    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" event={"ID":"144afe47-505e-436b-a07d-5e1ea36818a1","Type":"ContainerStarted","Data":"510e014c0873b6c61dade3316cdce23163efae99e590718a3e657d772dc47079"}
Oct 31 13:14:33.311983 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.311503    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" event={"ID":"144afe47-505e-436b-a07d-5e1ea36818a1","Type":"ContainerStarted","Data":"769e0f85c9adac4b22f2538a72747c4e700e733dbef0cac3d9d3cf14a99cc4b4"}
Oct 31 13:14:33.337071 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.337022    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podStartSLOduration=2.337005433 podStartE2EDuration="2.337005433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:33.330927108 +0000 UTC m=+852.746868945" watchObservedRunningTime="2026-10-31 13:14:33.337005433 +0000 UTC m=+852.752947273"
Oct 31 13:14:33.480932 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.480901    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:33.519732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.517082    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/744e9587-802d-495d-b30e-cec079e3f3f8-kube-api-access-d22v8\" (UniqueName: \"kubernetes.io/projected/744e9587-802d-495d-b30e-cec079e3f3f8-kube-api-access-d22v8\") pod \"744e9587-802d-495d-b30e-cec079e3f3f8\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") "
Oct 31 13:14:33.519732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.517351    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-openshift-ca\") pod \"744e9587-802d-495d-b30e-cec079e3f3f8\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") "
Oct 31 13:14:33.519732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.517511    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-xks-ca\") pod \"744e9587-802d-495d-b30e-cec079e3f3f8\" (UID: \"744e9587-802d-495d-b30e-cec079e3f3f8\") "
Oct 31 13:14:33.545731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.532763    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-openshift-ca" pod "744e9587-802d-495d-b30e-cec079e3f3f8" (UID: "744e9587-802d-495d-b30e-cec079e3f3f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:14:33.588420 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.587414    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-xks-ca" pod "744e9587-802d-495d-b30e-cec079e3f3f8" (UID: "744e9587-802d-495d-b30e-cec079e3f3f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:14:33.594504 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.594421    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/744e9587-802d-495d-b30e-cec079e3f3f8-kube-api-access-d22v8" pod "744e9587-802d-495d-b30e-cec079e3f3f8" (UID: "744e9587-802d-495d-b30e-cec079e3f3f8"). InnerVolumeSpecName "kube-api-access-d22v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:14:33.631617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.631244    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:33.631875 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.631851    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/744e9587-802d-495d-b30e-cec079e3f3f8-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:33.632011 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.631996    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d22v8\" (UniqueName: \"kubernetes.io/projected/744e9587-802d-495d-b30e-cec079e3f3f8-kube-api-access-d22v8\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:33.917303 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:33.917228    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:34.320561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:34.320524    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62" event={"ID":"744e9587-802d-495d-b30e-cec079e3f3f8","Type":"ContainerDied","Data":"ad3f8633e7d3f9e5d8d7b5192388f4dd9f15a4f12d8f48580d6c6dbdc1c230c9"}
Oct 31 13:14:34.320561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:34.320550    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"
Oct 31 13:14:34.321088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:34.320595    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad3f8633e7d3f9e5d8d7b5192388f4dd9f15a4f12d8f48580d6c6dbdc1c230c9"
Oct 31 13:14:35.016104 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.016066    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"]
Oct 31 13:14:35.016642 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.016629    5787 cpu_manager.go:414] "containerMap: removing container" podUID="744e9587-802d-495d-b30e-cec079e3f3f8" containerName="revoke-keys"
Oct 31 13:14:35.016734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.016643    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="744e9587-802d-495d-b30e-cec079e3f3f8" containerName="revoke-keys"
Oct 31 13:14:35.016799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.016789    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="744e9587-802d-495d-b30e-cec079e3f3f8" containerName="revoke-keys"
Oct 31 13:14:35.020659 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.020641    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.027289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.027261    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:14:35.027477 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.027285    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:14:35.027477 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.027352    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:14:35.028610 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.027357    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:14:35.032792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.032770    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"]
Oct 31 13:14:35.157601 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.157563    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mgql\" (UniqueName: \"kubernetes.io/projected/875d6f05-6e36-4c65-bea2-11c71f1997bb-kube-api-access-6mgql\") pod \"maas-api-revoke-sub-ddf55a7f-zxsj7\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.157827 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.157632    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddf55a7f-zxsj7\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.157993 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.157969    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddf55a7f-zxsj7\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.258995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.258953    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mgql\" (UniqueName: \"kubernetes.io/projected/875d6f05-6e36-4c65-bea2-11c71f1997bb-kube-api-access-6mgql\") pod \"maas-api-revoke-sub-ddf55a7f-zxsj7\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.259201 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.259052    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddf55a7f-zxsj7\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.259310 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.259239    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddf55a7f-zxsj7\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.259992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.259970    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ddf55a7f-zxsj7\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.261861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.261834    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ddf55a7f-zxsj7\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.275582 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.275516    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mgql\" (UniqueName: \"kubernetes.io/projected/875d6f05-6e36-4c65-bea2-11c71f1997bb-kube-api-access-6mgql\") pod \"maas-api-revoke-sub-ddf55a7f-zxsj7\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.337920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.337881    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:35.638583 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:35.638547    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"]
Oct 31 13:14:35.639232 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:35.639203    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875d6f05_6e36_4c65_bea2_11c71f1997bb.slice/crio-992b161cd785c7f38a355d55b03cbf9faffc385e456290e591fdb2ce58a87351 WatchSource:0}: Error finding container 992b161cd785c7f38a355d55b03cbf9faffc385e456290e591fdb2ce58a87351: Status 404 returned error can't find the container with id 992b161cd785c7f38a355d55b03cbf9faffc385e456290e591fdb2ce58a87351
Oct 31 13:14:36.340187 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.340094    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7" event={"ID":"875d6f05-6e36-4c65-bea2-11c71f1997bb","Type":"ContainerStarted","Data":"12c87682066575bb372f6c1549538397966721dd311fa692834dd0d415a727c8"}
Oct 31 13:14:36.340187 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.340152    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7" event={"ID":"875d6f05-6e36-4c65-bea2-11c71f1997bb","Type":"ContainerStarted","Data":"992b161cd785c7f38a355d55b03cbf9faffc385e456290e591fdb2ce58a87351"}
Oct 31 13:14:36.366513 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.366433    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7" podStartSLOduration=2.366415897 podStartE2EDuration="2.366415897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:36.364982201 +0000 UTC m=+855.780924041" watchObservedRunningTime="2026-10-31 13:14:36.366415897 +0000 UTC m=+855.782357739"
Oct 31 13:14:36.729074 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.729036    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"]
Oct 31 13:14:36.737326 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.737289    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"]
Oct 31 13:14:36.737581 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.737553    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.812607 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.812567    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.812836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.812629    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.812836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.812791    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.812933 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.812868    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.812992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.812974    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.813042 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.813026    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72d85\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-kube-api-access-72d85\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.813250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.813237    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.813295 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.813280    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.813381 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.813362    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.814684 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.814662    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"]
Oct 31 13:14:36.819084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.819061    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.837086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.837053    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"]
Oct 31 13:14:36.900037 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.900001    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"]
Oct 31 13:14:36.904572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.904544    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:36.915200 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.915170    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.915347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.915225    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.915347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.915306    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.915468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.915431    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.915526 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.915477    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.915577 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.915554    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72d85\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-kube-api-access-72d85\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.915628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.915591    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.915879 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.915851    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.915970 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.915953    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.916090 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916004    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.916090 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916018    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-data\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.916220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916104    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.916220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916144    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e7efa4-f7a0-4268-8356-91ef1caaf034-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.916220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916172    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.916324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916230    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.916324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916260    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.916324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916285    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85kj2\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-kube-api-access-85kj2\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:36.916324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916308    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.916449 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916336    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.916684 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.916661    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-certs\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.918852 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.917145    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.918956 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.917832    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istiod-ca-cert\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.918956 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.918063    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-credential-socket\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.919906 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.919881    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-envoy\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.922216 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.922194    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-podinfo\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.925409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.925382    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"]
Oct 31 13:14:36.939194 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.939159    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72d85\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-kube-api-access-72d85\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:36.942387 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:36.942359    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-token\") pod \"e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") " pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:37.017818 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.017723    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.017818 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.017770    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e7efa4-f7a0-4268-8356-91ef1caaf034-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.018122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.017878    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc7km\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-kube-api-access-qc7km\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.018122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.017947    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.018122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018027    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85kj2\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-kube-api-access-85kj2\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.018122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018091    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.018369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018142    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.018369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018200    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.018369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018234    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.018369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018256    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.018369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018291    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.018369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018328    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.018369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018364    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.018737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018419    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.018737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018490    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9206bfbe-392b-4a0e-972e-618cbafa13f9-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.018737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018531    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.018737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018560    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.018737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018605    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.019020 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018904    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-certs\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.019020 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.018604    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e7efa4-f7a0-4268-8356-91ef1caaf034-istiod-ca-cert\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.019409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.019379    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-data\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.019992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.019698    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.020087 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.020063    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-credential-socket\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.022240 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.022218    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-envoy\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.022545 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.022471    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-podinfo\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.037276 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.037248    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85kj2\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-kube-api-access-85kj2\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.042187 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.042083    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-token\") pod \"e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") " pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.054118 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.054086    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:37.121162 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.120921    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121162 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121001    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121162 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121073    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9206bfbe-392b-4a0e-972e-618cbafa13f9-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121162 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121121    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121395 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121200    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121395 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121284    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc7km\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-kube-api-access-qc7km\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121406    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121451    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121477    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121491    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.121835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.121815    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.123599 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.122846    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9206bfbe-392b-4a0e-972e-618cbafa13f9-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.123599 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.123559    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.124058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.124021    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.126599 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.126554    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.142335 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.142300    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc7km\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-kube-api-access-qc7km\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.156892 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.149759    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:37.157572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.157546    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.162058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.162027    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.223256 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.223217    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:37.352379 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.352339    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"]
Oct 31 13:14:37.360960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.360341    5787 generic.go:356] "Generic (PLEG): container finished" podID="875d6f05-6e36-4c65-bea2-11c71f1997bb" containerID="12c87682066575bb372f6c1549538397966721dd311fa692834dd0d415a727c8" exitCode=0
Oct 31 13:14:37.367772 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.367736    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7" event={"ID":"875d6f05-6e36-4c65-bea2-11c71f1997bb","Type":"ContainerDied","Data":"12c87682066575bb372f6c1549538397966721dd311fa692834dd0d415a727c8"}
Oct 31 13:14:37.368173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.368071    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.370354 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.370330    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"]
Oct 31 13:14:37.379341 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.379315    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"]
Oct 31 13:14:37.381943 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:37.381816    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a714d74_2bc2_4eb2_b3f4_5471047b8cf5.slice/crio-0ffbf2cca149488255ac77f7833925e23ec14d519beeb873fee4c30055bc4dcb WatchSource:0}: Error finding container 0ffbf2cca149488255ac77f7833925e23ec14d519beeb873fee4c30055bc4dcb: Status 404 returned error can't find the container with id 0ffbf2cca149488255ac77f7833925e23ec14d519beeb873fee4c30055bc4dcb
Oct 31 13:14:37.384407 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.384365    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:37.384520 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.384454    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:37.384520 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.384506    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:37.428688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.426764    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.428688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.427880    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.428688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.427921    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.428688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.427975    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.428688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.428050    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.428688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.428092    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b687cd60-1308-468e-a24f-674c6cef2f8a-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.428688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.428409    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvs5w\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-kube-api-access-xvs5w\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.428688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.428433    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.428688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.428529    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.467279 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.467252    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"]
Oct 31 13:14:37.481203 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:37.481158    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e7efa4_f7a0_4268_8356_91ef1caaf034.slice/crio-1743fb06bfed1565d8a97e4420eb9ca100279624e07b99b9b730be6df64bf4a0 WatchSource:0}: Error finding container 1743fb06bfed1565d8a97e4420eb9ca100279624e07b99b9b730be6df64bf4a0: Status 404 returned error can't find the container with id 1743fb06bfed1565d8a97e4420eb9ca100279624e07b99b9b730be6df64bf4a0
Oct 31 13:14:37.484011 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.483973    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:37.484119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.484067    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:37.484119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.484102    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:37.541014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.540347    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b687cd60-1308-468e-a24f-674c6cef2f8a-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.543377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.541453    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvs5w\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-kube-api-access-xvs5w\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.543377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.541521    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.543377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.541625    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.543377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.541871    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.543377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.541913    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.543377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.541939    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.543377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.541992    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.543377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.542357    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.543377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.542925    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-certs\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.550479 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.544463    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b687cd60-1308-468e-a24f-674c6cef2f8a-istiod-ca-cert\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.551350 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.551292    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-data\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.551950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.551346    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.552390 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.552353    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-credential-socket\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.559096 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.558813    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-podinfo\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.561074 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.561042    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-envoy\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.567329 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.567251    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvs5w\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-kube-api-access-xvs5w\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.582083 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.581947    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-token\") pod \"e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") " pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:37.645824 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.625263    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:37.645824 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.625360    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:37.645824 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.625402    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:37.645824 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.625671    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"]
Oct 31 13:14:37.713482 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:37.712144    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:38.149417 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:38.149372    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb687cd60_1308_468e_a24f_674c6cef2f8a.slice/crio-cf9cf04566f9d9bce273dc89f5ed16b572ff3f8a6dd45e15462d9b4614bc14cc WatchSource:0}: Error finding container cf9cf04566f9d9bce273dc89f5ed16b572ff3f8a6dd45e15462d9b4614bc14cc: Status 404 returned error can't find the container with id cf9cf04566f9d9bce273dc89f5ed16b572ff3f8a6dd45e15462d9b4614bc14cc
Oct 31 13:14:38.150861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.150775    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"]
Oct 31 13:14:38.153748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.153694    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:38.153862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.153791    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:38.153862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.153821    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:14:38.381011 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.380971    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" event={"ID":"b687cd60-1308-468e-a24f-674c6cef2f8a","Type":"ContainerStarted","Data":"3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db"}
Oct 31 13:14:38.381583 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.381559    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" event={"ID":"b687cd60-1308-468e-a24f-674c6cef2f8a","Type":"ContainerStarted","Data":"cf9cf04566f9d9bce273dc89f5ed16b572ff3f8a6dd45e15462d9b4614bc14cc"}
Oct 31 13:14:38.384394 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.384357    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" event={"ID":"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5","Type":"ContainerStarted","Data":"7c722275a5ad1c43ead8c1809c21b33b8203245a13f4815e8d2bda7232e47263"}
Oct 31 13:14:38.384544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.384420    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" event={"ID":"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5","Type":"ContainerStarted","Data":"0ffbf2cca149488255ac77f7833925e23ec14d519beeb873fee4c30055bc4dcb"}
Oct 31 13:14:38.391207 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.391173    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" event={"ID":"9206bfbe-392b-4a0e-972e-618cbafa13f9","Type":"ContainerStarted","Data":"142c71f627c40f94389a904d16edd3f684c7ee28e9dfa1d2363b05e90d112fa9"}
Oct 31 13:14:38.391436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.391421    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" event={"ID":"9206bfbe-392b-4a0e-972e-618cbafa13f9","Type":"ContainerStarted","Data":"b6ab8e1bbb3e3a50d6c27b004207fa228d71a9b151694aa887b11a7838302683"}
Oct 31 13:14:38.393762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.393721    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" event={"ID":"94e7efa4-f7a0-4268-8356-91ef1caaf034","Type":"ContainerStarted","Data":"d8ef1b7043a1afc31c3ac72c40c3aa27744a7ea7276db0a8061c35bee2371ef5"}
Oct 31 13:14:38.393892 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.393779    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" event={"ID":"94e7efa4-f7a0-4268-8356-91ef1caaf034","Type":"ContainerStarted","Data":"1743fb06bfed1565d8a97e4420eb9ca100279624e07b99b9b730be6df64bf4a0"}
Oct 31 13:14:38.423821 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.422686    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podStartSLOduration=1.422665257 podStartE2EDuration="1.422665257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:38.411565495 +0000 UTC m=+857.827507338" watchObservedRunningTime="2026-10-31 13:14:38.422665257 +0000 UTC m=+857.838607097"
Oct 31 13:14:38.556727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.530316    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podStartSLOduration=2.530292037 podStartE2EDuration="2.530292037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:38.453045945 +0000 UTC m=+857.868987786" watchObservedRunningTime="2026-10-31 13:14:38.530292037 +0000 UTC m=+857.946233878"
Oct 31 13:14:38.556727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.530491    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podStartSLOduration=2.5304814 podStartE2EDuration="2.5304814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:38.497191478 +0000 UTC m=+857.913133341" watchObservedRunningTime="2026-10-31 13:14:38.5304814 +0000 UTC m=+857.946423242"
Oct 31 13:14:38.556727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.547299    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podStartSLOduration=2.547277865 podStartE2EDuration="2.547277865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:14:38.543514148 +0000 UTC m=+857.959455987" watchObservedRunningTime="2026-10-31 13:14:38.547277865 +0000 UTC m=+857.963219708"
Oct 31 13:14:38.718900 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:38.718813    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:39.057788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.056692    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:39.078684 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.078645    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:39.079019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.078967    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:39.182243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.182190    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:39.228689 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.228217    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:39.228689 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.228347    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:39.228689 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.228396    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:39.232168 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.232096    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:39.232384 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.232318    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:39.493323 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.493290    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:39.514757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.512213    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7" event={"ID":"875d6f05-6e36-4c65-bea2-11c71f1997bb","Type":"ContainerDied","Data":"992b161cd785c7f38a355d55b03cbf9faffc385e456290e591fdb2ce58a87351"}
Oct 31 13:14:39.514757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.512271    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="992b161cd785c7f38a355d55b03cbf9faffc385e456290e591fdb2ce58a87351"
Oct 31 13:14:39.623729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.622891    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-openshift-ca\") pod \"875d6f05-6e36-4c65-bea2-11c71f1997bb\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") "
Oct 31 13:14:39.623729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.622967    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-xks-ca\") pod \"875d6f05-6e36-4c65-bea2-11c71f1997bb\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") "
Oct 31 13:14:39.623729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.623061    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875d6f05-6e36-4c65-bea2-11c71f1997bb-kube-api-access-6mgql\" (UniqueName: \"kubernetes.io/projected/875d6f05-6e36-4c65-bea2-11c71f1997bb-kube-api-access-6mgql\") pod \"875d6f05-6e36-4c65-bea2-11c71f1997bb\" (UID: \"875d6f05-6e36-4c65-bea2-11c71f1997bb\") "
Oct 31 13:14:39.639743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.624692    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-openshift-ca" pod "875d6f05-6e36-4c65-bea2-11c71f1997bb" (UID: "875d6f05-6e36-4c65-bea2-11c71f1997bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:14:39.719544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.718801    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:39.719544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.718897    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:39.735734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.731741    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:39.748982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.748855    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-xks-ca" pod "875d6f05-6e36-4c65-bea2-11c71f1997bb" (UID: "875d6f05-6e36-4c65-bea2-11c71f1997bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:14:39.750949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.750896    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875d6f05-6e36-4c65-bea2-11c71f1997bb-kube-api-access-6mgql" pod "875d6f05-6e36-4c65-bea2-11c71f1997bb" (UID: "875d6f05-6e36-4c65-bea2-11c71f1997bb"). InnerVolumeSpecName "kube-api-access-6mgql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:14:39.836496 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.836446    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/875d6f05-6e36-4c65-bea2-11c71f1997bb-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:39.836790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.836767    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mgql\" (UniqueName: \"kubernetes.io/projected/875d6f05-6e36-4c65-bea2-11c71f1997bb-kube-api-access-6mgql\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:39.910998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:39.910954    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:40.057258 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.057156    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:40.057515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.057482    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:40.154587 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.154548    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:40.154894 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.154866    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:40.183850 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.183803    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:40.225735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.223808    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:40.225735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.225480    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:40.225735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.225526    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:40.285987 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.285938    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:40.559280 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.559237    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:40.609731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.589726    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:40.635838 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.635803    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"
Oct 31 13:14:40.724342 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.724311    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:40.724533 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.724507    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:40.911964 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.911859    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:40.912979 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.912956    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:40.936030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:40.935988    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:41.093730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.090852    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:41.093909 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.093887    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:41.093977 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.093935    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:41.189878 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.164647    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:41.189878 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.164730    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:41.227156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.227122    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:41.395857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.227410    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:41.395857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.263125    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:41.395857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.264050    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:14:41.395857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.239891    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:41.395857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.268281    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:14:41.395857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.295800    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:41.395857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.296277    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:14:41.395857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.349409    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:41.474222 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.472546    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:41.653739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.618016    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:14:41.653739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.649957    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 31 13:14:41.653739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.652656    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:14:41.653739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.652767    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:14:41.734786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.729326    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:41.735064 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.735028    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:41.945732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.937092    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:42.025678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:41.957899    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:42.035918 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.035883    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:14:42.072231 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.072196    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:42.072543 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.072513    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:42.180958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.180920    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:42.181241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.181205    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:42.252753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.252041    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:42.253172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.253143    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:42.253394 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.253369    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:42.335562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.335468    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:42.337032 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.337003    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:42.340770 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.340035    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:42.359915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.351813    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:42.362798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.360229    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:14:42.685851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.657319    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:14:42.721096 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.721033    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:42.747218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.747177    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:42.782732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.748350    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:42.782732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.748402    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:42.993788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:42.992815    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:43.052734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.028369    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.052734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.047875    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.077733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.056254    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:43.077733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.056307    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:43.178742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.157779    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:43.178742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.157843    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:43.273791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.244028    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:43.273791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.244087    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:43.337727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.336972    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.337727 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.337456    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.349817 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.340156    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.358550 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.356806    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.510910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.509758    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.525322 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.524811    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.658728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.657846    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.715732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.715203    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:43.715732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.715265    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:43.721842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.721796    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.722493 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.722472    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:14:43.753833 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.753793    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:43.769847 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:43.769800    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:44.033730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.029236    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:44.038491 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.036796    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:44.059048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.059013    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:44.078729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.059298    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:44.078729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.059634    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:44.078729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.059978    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:14:44.228830 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.205829    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:44.228830 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.216980    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:44.274735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.242545    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:44.275067 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.275030    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:44.414827 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.356097    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:44.414827 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.356570    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:44.436733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.386963    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:44.436733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.421395    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:44.436733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.428236    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:44.516533 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.516492    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:44.524745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.517603    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:44.675835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.660878    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:44.723734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.714116    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:44.731941 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.731887    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:44.736964 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.736920    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:44.877766 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:44.770533    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.086728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.086623    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:45.108786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.086682    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:45.108786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.105230    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 31 13:14:45.108786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.105281    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.108786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.105675    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.132731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.130940    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.153235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.153110    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:45.153235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.153172    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:45.199733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.178023    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:14:45.230619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.230582    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:45.478219 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.478085    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:45.478496 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.341622    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.479326 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.363835    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:45.480318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.363872    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:45.480867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.450327    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.486755 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.470722    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.491456 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.231972    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.492005 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.482818    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.525744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.524386    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:45.525916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.525842    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.527287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.526063    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.602741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.576900    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.732794 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.732563    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:45.732794 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.732636    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:45.762039 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.761991    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:45.906884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:45.906827    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:46.056265 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.056160    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:46.056521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.056491    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:46.188725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.180924    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:46.188725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.182135    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:46.188725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.182191    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:46.188725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.182552    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:46.206729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.196390    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:46.225152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.225121    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:46.225433 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.225410    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:46.297263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.297236    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:46.506887 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.506838    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:46.507642 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.507114    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:46.508388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.507142    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:46.514994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.514956    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:46.554728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.552205    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:46.554728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.552695    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:14:46.703726 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.695255    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:46.713083 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.713047    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:46.713347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.713324    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:46.733623 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.733582    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:46.769626 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.769530    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:46.968013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:46.967958    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 31 13:14:47.069812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.068096    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:47.069812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.068159    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:47.227864 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.227803    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:47.244788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.244750    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:47.255848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.255807    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:47.256082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.247903    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:47.256437 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.246136    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 31 13:14:47.256691 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.246410    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:47.256898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.256873    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:47.351975 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.351853    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:47.393731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.389901    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:47.511058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.511003    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:47.511682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.511649    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:47.512810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.512775    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 31 13:14:47.526422 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.526375    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:47.562377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.562331    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:47.563451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.563422    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 31 13:14:47.695865 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.695810    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:47.696879 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.696847    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:14:47.716194 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.716160    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:47.716488 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.716453    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:47.915149 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.915089    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:47.973847 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:47.972966    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:48.048854 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.048804    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:48.055469 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.055435    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:48.055796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.055747    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:48.150556 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.150517    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:48.150870 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.150841    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:48.203249 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.203190    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:48.241018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.238729    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:48.241018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.239357    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:48.258802 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.256239    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/health\": context deadline exceeded"
Oct 31 13:14:48.258802 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.256363    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:48.258802 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.257727    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:48.282849 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.282813    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:48.283147 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.283119    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:48.415736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.372492    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:14:48.415736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.372951    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:48.518937 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.518825    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:48.519580 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.519556    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:48.552733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.546282    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:48.632808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.547724    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:48.633888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.547763    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:48.634402 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.576219    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:48.653761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.602420    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:48.713851 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:14:48.713802    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:14:48.796393 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.796294    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:14:48.809078 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.809043    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:48.809327 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.809298    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:48.916773 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.915419    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:48.917573 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:48.917537    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:49.078017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.075785    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:49.078017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.075844    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:49.229769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.222819    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 31 13:14:49.229769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.223750    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:49.229769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.223792    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:49.229769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.225332    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:49.229769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.225394    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:49.270914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.270862    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:49.271167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.270876    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:49.391419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.391320    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:49.529182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.529135    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:49.739388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.739350    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:49.760064 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.760020    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:49.760292 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.741488    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:49.760786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.741451    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:49.761531 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.741511    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:49.768213 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.741532    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 31 13:14:49.768664 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.757434    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:49.798843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:49.798796    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:50.152784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.152662    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:50.235015 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.232942    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 31 13:14:50.235015 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.233012    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 31 13:14:50.242748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.235340    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:50.242748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.236344    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:50.242748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.236383    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:50.242748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.237414    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:50.242748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.237447    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:50.278255 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.277326    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:50.292761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.282799    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:50.505731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.505500    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:50.530897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.530840    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 31 13:14:50.944277 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.944235    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:50.960019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.946792    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:50.962864 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.946920    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:50.983858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.965353    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:50.983858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.969916    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 31 13:14:50.983858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.971270    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:50.984900 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.984846    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:50.985126 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:50.985083    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:51.105911 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.105821    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Oct 31 13:14:51.106245 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.106184    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Oct 31 13:14:51.176780 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.176689    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:51.180525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.180480    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:51.292796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.249173    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:51.379179 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.318384    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:51.379179 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.327918    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:51.379179 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.371367    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:51.379179 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.371407    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:51.466603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.466563    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"]
Oct 31 13:14:51.541859 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.508242    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:51.542470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.537134    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 31 13:14:51.569782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.567225    5787 cpu_manager.go:414] "containerMap: removing container" podUID="875d6f05-6e36-4c65-bea2-11c71f1997bb" containerName="revoke-keys"
Oct 31 13:14:51.569782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.567260    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875d6f05-6e36-4c65-bea2-11c71f1997bb" containerName="revoke-keys"
Oct 31 13:14:51.569782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.567484    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="875d6f05-6e36-4c65-bea2-11c71f1997bb" containerName="revoke-keys"
Oct 31 13:14:51.596952 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.596919    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"]
Oct 31 13:14:51.597304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.597284    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.630731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.628463    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:14:51.630731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.628639    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:14:51.630731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.629531    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:14:51.656310 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.653915    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:14:51.692808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.667400    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftpsq\" (UniqueName: \"kubernetes.io/projected/845451b4-9361-4c2b-a593-b7a68b6f6362-kube-api-access-ftpsq\") pod \"maas-api-revoke-sub-804d78ab-rhhww\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.693050 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.693029    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-804d78ab-rhhww\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.693222 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.693202    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-804d78ab-rhhww\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.717744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.717692    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:51.719515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.719483    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:51.801055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.801008    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-804d78ab-rhhww\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.811883 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.811849    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftpsq\" (UniqueName: \"kubernetes.io/projected/845451b4-9361-4c2b-a593-b7a68b6f6362-kube-api-access-ftpsq\") pod \"maas-api-revoke-sub-804d78ab-rhhww\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.848786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.836474    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-804d78ab-rhhww\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.849183 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.820303    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-xks-ca\") pod \"maas-api-revoke-sub-804d78ab-rhhww\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.865739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.865172    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-804d78ab-rhhww\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.868742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.865934    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftpsq\" (UniqueName: \"kubernetes.io/projected/845451b4-9361-4c2b-a593-b7a68b6f6362-kube-api-access-ftpsq\") pod \"maas-api-revoke-sub-804d78ab-rhhww\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:51.950852 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.950800    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:51.975500 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.975452    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:51.976169 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.976139    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:51.976591 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.976568    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 31 13:14:51.977378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.977356    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:51.977812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:51.977507    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:52.056204 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.056163    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:52.058026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.056821    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:52.154865 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.154831    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:52.180882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.180822    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:52.210166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.187175    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:52.229733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.211265    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:52.231728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.231684    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:52.248958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.231258    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:52.248958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.232239    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:52.248958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.231611    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Oct 31 13:14:52.248958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.232532    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Oct 31 13:14:52.271734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.265175    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:52.272049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.272011    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:52.457209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.376839    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:52.457209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.379787    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:52.457209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.380181    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 31 13:14:52.554624 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.552799    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:14:52.554624 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.554001    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:52.720790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.720169    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:52.720790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.720230    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:52.896842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.896800    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:52.897110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.897078    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:52.915061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.915029    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:52.938121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.937831    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:52.964956 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.964909    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.004792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.976796    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:53.004792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.977199    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:14:53.004792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.978304    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.004792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:52.978565    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:53.007372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.006359    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded"
Oct 31 13:14:53.107934 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.107893    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:53.108197 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.108167    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.211958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.211801    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.218261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.218210    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:53.351696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.284822    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:53.351696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.284903    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.402204 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.399503    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.411375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.411336    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.414731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.411608    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.635217 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.634794    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.647764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.635537    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.784964 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.784914    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Oct 31 13:14:53.785271 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.785226    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Oct 31 13:14:53.823732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.792795    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/health\": context deadline exceeded"
Oct 31 13:14:53.853886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.853839    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.978590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.919297    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"]
Oct 31 13:14:53.978926 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.974053    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:53.992734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.977873    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:54.198299 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.977926    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:14:54.281238 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.986542    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:54.288107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:53.986576    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:54.295853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.014409    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:54.305483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.211910    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:54.305756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.305723    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:54.306416 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.279298    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:14:54.306609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.306595    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:14:54.335171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.331390    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:54.335171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.331446    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:54.363693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.348951    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:54.363693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.349039    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:54.363693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.361908    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:54.411832 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.411788    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:54.477731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.477573    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:14:54.636224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.636097    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:54.679651 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.679592    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:14:54.886511 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.684396    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 31 13:14:54.887139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.688776    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:54.889561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:54.688813    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.262865 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.253921    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded"
Oct 31 13:14:55.262865 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.255147    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.282201 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.282156    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.381870 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.295937    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.436592 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.296068    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:14:55.560388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.560281    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 31 13:14:55.637729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.390178    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:55.638113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.638057    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.638464 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.608480    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.638694 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.608530    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:55.639160 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.369864    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:55.639320 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.446756    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:55.658768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.639435    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.659266 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.659233    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:55.659541 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.659513    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.737734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.702132    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:14:55.798750 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.787010    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:14:55.862937 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.862835    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.889724 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.873773    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:55.889724 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.873840    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.901744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.899380    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.901744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.899756    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:55.955956 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.955925    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:55.976072 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:55.975988    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:14:56.233346 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.233313    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:56.324411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.282654    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:56.354159 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.351852    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:14:56.372931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.372881    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:56.406345 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.406301    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww" event={"ID":"845451b4-9361-4c2b-a593-b7a68b6f6362","Type":"ContainerStarted","Data":"0d977fb9fa7a875031ab9e41b042816addb45fefe40fe383e2ce5438b4581c12"}
Oct 31 13:14:56.450918 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.450401    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:56.465746 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.465685    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:56.488758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.488665    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:14:56.658739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.645835    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:14:56.658739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.648935    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Oct 31 13:14:56.662390 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.662022    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 31 13:14:56.691315 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.690967    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:56.691635 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.691585    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:56.693734 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.693683    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:14:56.714588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.713849    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:14:56.714588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.713910    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:56.818652 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.818620    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:56.819039 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.819019    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:14:56.819237 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.819213    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:14:56.822516 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.822494    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:14:56.906760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.855394    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:14:56.906760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.899899    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:14:56.941537 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:56.941155    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:14:57.025790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.016033    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:14:57.034376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.029747    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"]
Oct 31 13:14:57.150732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.083074    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.150732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.120874    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"]
Oct 31 13:14:57.227731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.190847    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgdlr\" (UniqueName: \"kubernetes.io/projected/800893dc-ceaa-4010-87f6-4eaba5bc5aef-kube-api-access-tgdlr\") pod \"maas-api-revoke-sub-745b7bde-t5frp\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.227731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.191204    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-745b7bde-t5frp\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.227731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.191239    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-745b7bde-t5frp\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.317801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.293993    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-745b7bde-t5frp\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.317801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.294047    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-745b7bde-t5frp\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.317801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.294154    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgdlr\" (UniqueName: \"kubernetes.io/projected/800893dc-ceaa-4010-87f6-4eaba5bc5aef-kube-api-access-tgdlr\") pod \"maas-api-revoke-sub-745b7bde-t5frp\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.317801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.295504    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-745b7bde-t5frp\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.556891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.539052    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgdlr\" (UniqueName: \"kubernetes.io/projected/800893dc-ceaa-4010-87f6-4eaba5bc5aef-kube-api-access-tgdlr\") pod \"maas-api-revoke-sub-745b7bde-t5frp\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.556891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.545416    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-745b7bde-t5frp\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:57.607624 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.596756    5787 generic.go:356] "Generic (PLEG): container finished" podID="845451b4-9361-4c2b-a593-b7a68b6f6362" containerID="9f30a3db4fbe27c09e662bd697da5adaa4ded1b0d6329f05ba8ac928f743f823" exitCode=0
Oct 31 13:14:57.618781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.615601    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww" event={"ID":"845451b4-9361-4c2b-a593-b7a68b6f6362","Type":"ContainerDied","Data":"9f30a3db4fbe27c09e662bd697da5adaa4ded1b0d6329f05ba8ac928f743f823"}
Oct 31 13:14:57.661732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.661463    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:57.681365 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.679418    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:14:57.835279 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:57.835254    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:14:58.337171 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:14:58.337136    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod800893dc_ceaa_4010_87f6_4eaba5bc5aef.slice/crio-b044f71283d54b4ebaeb00a87598dd5df54412f85fafc3028570da1a8cb6264c WatchSource:0}: Error finding container b044f71283d54b4ebaeb00a87598dd5df54412f85fafc3028570da1a8cb6264c: Status 404 returned error can't find the container with id b044f71283d54b4ebaeb00a87598dd5df54412f85fafc3028570da1a8cb6264c
Oct 31 13:14:58.337325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.337181    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"]
Oct 31 13:14:58.605893 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.605812    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp" event={"ID":"800893dc-ceaa-4010-87f6-4eaba5bc5aef","Type":"ContainerStarted","Data":"b044f71283d54b4ebaeb00a87598dd5df54412f85fafc3028570da1a8cb6264c"}
Oct 31 13:14:58.776178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.776150    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:58.799924 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.799896    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-xks-ca\") pod \"845451b4-9361-4c2b-a593-b7a68b6f6362\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") "
Oct 31 13:14:58.800118 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.799957    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-openshift-ca\") pod \"845451b4-9361-4c2b-a593-b7a68b6f6362\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") "
Oct 31 13:14:58.800188 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.800161    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/845451b4-9361-4c2b-a593-b7a68b6f6362-kube-api-access-ftpsq\" (UniqueName: \"kubernetes.io/projected/845451b4-9361-4c2b-a593-b7a68b6f6362-kube-api-access-ftpsq\") pod \"845451b4-9361-4c2b-a593-b7a68b6f6362\" (UID: \"845451b4-9361-4c2b-a593-b7a68b6f6362\") "
Oct 31 13:14:58.800955 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.800575    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-openshift-ca" pod "845451b4-9361-4c2b-a593-b7a68b6f6362" (UID: "845451b4-9361-4c2b-a593-b7a68b6f6362"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:14:58.800955 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.800923    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:58.803640 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.803615    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-xks-ca" pod "845451b4-9361-4c2b-a593-b7a68b6f6362" (UID: "845451b4-9361-4c2b-a593-b7a68b6f6362"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:14:58.809916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.809875    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/845451b4-9361-4c2b-a593-b7a68b6f6362-kube-api-access-ftpsq" pod "845451b4-9361-4c2b-a593-b7a68b6f6362" (UID: "845451b4-9361-4c2b-a593-b7a68b6f6362"). InnerVolumeSpecName "kube-api-access-ftpsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:14:58.902409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.902333    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/845451b4-9361-4c2b-a593-b7a68b6f6362-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:58.902409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:58.902362    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftpsq\" (UniqueName: \"kubernetes.io/projected/845451b4-9361-4c2b-a593-b7a68b6f6362-kube-api-access-ftpsq\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:14:59.615039 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:59.614951    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"
Oct 31 13:14:59.615451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:59.614950    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww" event={"ID":"845451b4-9361-4c2b-a593-b7a68b6f6362","Type":"ContainerDied","Data":"0d977fb9fa7a875031ab9e41b042816addb45fefe40fe383e2ce5438b4581c12"}
Oct 31 13:14:59.615451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:59.615098    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d977fb9fa7a875031ab9e41b042816addb45fefe40fe383e2ce5438b4581c12"
Oct 31 13:14:59.616837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:59.616812    5787 generic.go:356] "Generic (PLEG): container finished" podID="800893dc-ceaa-4010-87f6-4eaba5bc5aef" containerID="fcde411c9a378442af3626a800095f2c9e69f260bd1551ef7fbcde30f307c6cf" exitCode=0
Oct 31 13:14:59.616992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:14:59.616867    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp" event={"ID":"800893dc-ceaa-4010-87f6-4eaba5bc5aef","Type":"ContainerDied","Data":"fcde411c9a378442af3626a800095f2c9e69f260bd1551ef7fbcde30f307c6cf"}
Oct 31 13:15:00.157740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.157688    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"]
Oct 31 13:15:00.158696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.158673    5787 cpu_manager.go:414] "containerMap: removing container" podUID="845451b4-9361-4c2b-a593-b7a68b6f6362" containerName="revoke-keys"
Oct 31 13:15:00.158696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.158696    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="845451b4-9361-4c2b-a593-b7a68b6f6362" containerName="revoke-keys"
Oct 31 13:15:00.158889 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.158876    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="845451b4-9361-4c2b-a593-b7a68b6f6362" containerName="revoke-keys"
Oct 31 13:15:00.165855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.164498    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:00.174997 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.174624    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"]
Oct 31 13:15:00.205375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.205326    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"]
Oct 31 13:15:00.205940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.205914    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:00.210450 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.210416    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"]
Oct 31 13:15:00.210560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.210460    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"]
Oct 31 13:15:00.210560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.210541    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:00.215110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.215081    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"]
Oct 31 13:15:00.221246 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.221223    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"]
Oct 31 13:15:00.227524 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.226981    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:00.228764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.228732    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbq8h\" (UniqueName: \"kubernetes.io/projected/f4b8769e-6fce-4fbd-8765-d7144d767d1c-kube-api-access-lbq8h\") pod \"maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8\" (UID: \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:00.228941 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.228847    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6t8g\" (UniqueName: \"kubernetes.io/projected/52986769-9ebe-449b-8e8b-1b1e82579282-kube-api-access-l6t8g\") pod \"maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l\" (UID: \"52986769-9ebe-449b-8e8b-1b1e82579282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:00.229441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.229058    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw\" (UID: \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:00.229441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.229125    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnz88\" (UniqueName: \"kubernetes.io/projected/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-kube-api-access-wnz88\") pod \"maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw\" (UID: \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:00.229441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.229193    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52986769-9ebe-449b-8e8b-1b1e82579282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l\" (UID: \"52986769-9ebe-449b-8e8b-1b1e82579282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:00.229441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.229221    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b8769e-6fce-4fbd-8765-d7144d767d1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8\" (UID: \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:00.235346 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.235316    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"]
Oct 31 13:15:00.330296 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.330260    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnz88\" (UniqueName: \"kubernetes.io/projected/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-kube-api-access-wnz88\") pod \"maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw\" (UID: \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:00.330495 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.330316    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52986769-9ebe-449b-8e8b-1b1e82579282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l\" (UID: \"52986769-9ebe-449b-8e8b-1b1e82579282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:00.330495 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.330339    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b8769e-6fce-4fbd-8765-d7144d767d1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8\" (UID: \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:00.330495 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.330360    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mnqd\" (UniqueName: \"kubernetes.io/projected/3736d19a-1161-42bc-bf98-ec9d6408a563-kube-api-access-5mnqd\") pod \"maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd\" (UID: \"3736d19a-1161-42bc-bf98-ec9d6408a563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:00.330658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.330515    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbq8h\" (UniqueName: \"kubernetes.io/projected/f4b8769e-6fce-4fbd-8765-d7144d767d1c-kube-api-access-lbq8h\") pod \"maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8\" (UID: \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:00.330658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.330625    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736d19a-1161-42bc-bf98-ec9d6408a563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd\" (UID: \"3736d19a-1161-42bc-bf98-ec9d6408a563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:00.330658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.330647    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6t8g\" (UniqueName: \"kubernetes.io/projected/52986769-9ebe-449b-8e8b-1b1e82579282-kube-api-access-l6t8g\") pod \"maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l\" (UID: \"52986769-9ebe-449b-8e8b-1b1e82579282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:00.330884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.330853    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw\" (UID: \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:00.331144 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.331118    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b8769e-6fce-4fbd-8765-d7144d767d1c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8\" (UID: \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:00.331240 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.331217    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52986769-9ebe-449b-8e8b-1b1e82579282-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l\" (UID: \"52986769-9ebe-449b-8e8b-1b1e82579282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:00.331441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.331423    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw\" (UID: \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:00.344546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.344515    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbq8h\" (UniqueName: \"kubernetes.io/projected/f4b8769e-6fce-4fbd-8765-d7144d767d1c-kube-api-access-lbq8h\") pod \"maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8\" (UID: \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:00.344696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.344672    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6t8g\" (UniqueName: \"kubernetes.io/projected/52986769-9ebe-449b-8e8b-1b1e82579282-kube-api-access-l6t8g\") pod \"maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l\" (UID: \"52986769-9ebe-449b-8e8b-1b1e82579282\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:00.346355 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.346334    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnz88\" (UniqueName: \"kubernetes.io/projected/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-kube-api-access-wnz88\") pod \"maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw\" (UID: \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:00.432782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.432666    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mnqd\" (UniqueName: \"kubernetes.io/projected/3736d19a-1161-42bc-bf98-ec9d6408a563-kube-api-access-5mnqd\") pod \"maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd\" (UID: \"3736d19a-1161-42bc-bf98-ec9d6408a563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:00.432944 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.432836    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736d19a-1161-42bc-bf98-ec9d6408a563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd\" (UID: \"3736d19a-1161-42bc-bf98-ec9d6408a563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:00.433455 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.433435    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736d19a-1161-42bc-bf98-ec9d6408a563-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd\" (UID: \"3736d19a-1161-42bc-bf98-ec9d6408a563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:00.445972 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.445937    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mnqd\" (UniqueName: \"kubernetes.io/projected/3736d19a-1161-42bc-bf98-ec9d6408a563-kube-api-access-5mnqd\") pod \"maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd\" (UID: \"3736d19a-1161-42bc-bf98-ec9d6408a563\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:00.503017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.502981    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:00.532435 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.532401    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:00.538853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.538826    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:00.547943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.547919    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:00.832492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.829969    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"]
Oct 31 13:15:00.840268 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:00.840225    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b8769e_6fce_4fbd_8765_d7144d767d1c.slice/crio-3b55af145e59cd652196ff28c1d6479803889899c59f7be4ae5550ea78fe62f3 WatchSource:0}: Error finding container 3b55af145e59cd652196ff28c1d6479803889899c59f7be4ae5550ea78fe62f3: Status 404 returned error can't find the container with id 3b55af145e59cd652196ff28c1d6479803889899c59f7be4ae5550ea78fe62f3
Oct 31 13:15:00.941438 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:00.941408    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52986769_9ebe_449b_8e8b_1b1e82579282.slice/crio-46d22a0ad43be3ab13d82057b3ba84c63a6e14fd8bfbb9fdb233d9ef34c8d4f4 WatchSource:0}: Error finding container 46d22a0ad43be3ab13d82057b3ba84c63a6e14fd8bfbb9fdb233d9ef34c8d4f4: Status 404 returned error can't find the container with id 46d22a0ad43be3ab13d82057b3ba84c63a6e14fd8bfbb9fdb233d9ef34c8d4f4
Oct 31 13:15:00.946686 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:00.946630    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"]
Oct 31 13:15:01.029033 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:01.029008    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e1fa248_4cdb_4a4d_99e4_d9a259547d84.slice/crio-9cb8a80d58b0646091f3e8a3a0188ef51a44d0edc39244b433519226e97835f6 WatchSource:0}: Error finding container 9cb8a80d58b0646091f3e8a3a0188ef51a44d0edc39244b433519226e97835f6: Status 404 returned error can't find the container with id 9cb8a80d58b0646091f3e8a3a0188ef51a44d0edc39244b433519226e97835f6
Oct 31 13:15:01.029796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.029688    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"]
Oct 31 13:15:01.041180 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.041154    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:15:01.050888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.050852    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/800893dc-ceaa-4010-87f6-4eaba5bc5aef-kube-api-access-tgdlr\" (UniqueName: \"kubernetes.io/projected/800893dc-ceaa-4010-87f6-4eaba5bc5aef-kube-api-access-tgdlr\") pod \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") "
Oct 31 13:15:01.051091 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.051072    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-openshift-ca\") pod \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") "
Oct 31 13:15:01.051189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.051172    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-xks-ca\") pod \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\" (UID: \"800893dc-ceaa-4010-87f6-4eaba5bc5aef\") "
Oct 31 13:15:01.051472 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.051442    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-openshift-ca" pod "800893dc-ceaa-4010-87f6-4eaba5bc5aef" (UID: "800893dc-ceaa-4010-87f6-4eaba5bc5aef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:01.052129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.052110    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:01.068349 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.068324    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"]
Oct 31 13:15:01.134866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.134802    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/800893dc-ceaa-4010-87f6-4eaba5bc5aef-kube-api-access-tgdlr" pod "800893dc-ceaa-4010-87f6-4eaba5bc5aef" (UID: "800893dc-ceaa-4010-87f6-4eaba5bc5aef"). InnerVolumeSpecName "kube-api-access-tgdlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:01.134866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.134848    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-xks-ca" pod "800893dc-ceaa-4010-87f6-4eaba5bc5aef" (UID: "800893dc-ceaa-4010-87f6-4eaba5bc5aef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:01.136848 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:01.136824    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3736d19a_1161_42bc_bf98_ec9d6408a563.slice/crio-ad4cbaf13704f790376ce167ba5698240651f80f801d5869181077dabd35d9d1 WatchSource:0}: Error finding container ad4cbaf13704f790376ce167ba5698240651f80f801d5869181077dabd35d9d1: Status 404 returned error can't find the container with id ad4cbaf13704f790376ce167ba5698240651f80f801d5869181077dabd35d9d1
Oct 31 13:15:01.153356 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.153328    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/800893dc-ceaa-4010-87f6-4eaba5bc5aef-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:01.153356 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.153357    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgdlr\" (UniqueName: \"kubernetes.io/projected/800893dc-ceaa-4010-87f6-4eaba5bc5aef-kube-api-access-tgdlr\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:01.585562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.585528    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"]
Oct 31 13:15:01.586235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.586222    5787 cpu_manager.go:414] "containerMap: removing container" podUID="800893dc-ceaa-4010-87f6-4eaba5bc5aef" containerName="revoke-keys"
Oct 31 13:15:01.586306 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.586237    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="800893dc-ceaa-4010-87f6-4eaba5bc5aef" containerName="revoke-keys"
Oct 31 13:15:01.586347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.586338    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="800893dc-ceaa-4010-87f6-4eaba5bc5aef" containerName="revoke-keys"
Oct 31 13:15:01.590651 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.590630    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.600689 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.600662    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"]
Oct 31 13:15:01.660508 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.660472    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.660686 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.660672    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.660826 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.660807    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5p97\" (UniqueName: \"kubernetes.io/projected/78ea089d-9578-4ea5-9361-0d0452da506d-kube-api-access-d5p97\") pod \"maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.762126 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.762092    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.762278 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.762259    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.762369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.762353    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5p97\" (UniqueName: \"kubernetes.io/projected/78ea089d-9578-4ea5-9361-0d0452da506d-kube-api-access-d5p97\") pod \"maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.762965 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.762938    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.764789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.764767    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.844730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.832945    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5p97\" (UniqueName: \"kubernetes.io/projected/78ea089d-9578-4ea5-9361-0d0452da506d-kube-api-access-d5p97\") pod \"maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.879981 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.879875    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" event={"ID":"3736d19a-1161-42bc-bf98-ec9d6408a563","Type":"ContainerStarted","Data":"7c24bac43936dcb8a6d30ed26d6b433068ddfc78756a17d1ef2cd0e386afd1e1"}
Oct 31 13:15:01.879981 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.879942    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" event={"ID":"3736d19a-1161-42bc-bf98-ec9d6408a563","Type":"ContainerStarted","Data":"ad4cbaf13704f790376ce167ba5698240651f80f801d5869181077dabd35d9d1"}
Oct 31 13:15:01.884742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.883551    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp" event={"ID":"800893dc-ceaa-4010-87f6-4eaba5bc5aef","Type":"ContainerDied","Data":"b044f71283d54b4ebaeb00a87598dd5df54412f85fafc3028570da1a8cb6264c"}
Oct 31 13:15:01.884742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.883614    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b044f71283d54b4ebaeb00a87598dd5df54412f85fafc3028570da1a8cb6264c"
Oct 31 13:15:01.884742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.883732    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"
Oct 31 13:15:01.891421 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.890383    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" event={"ID":"f4b8769e-6fce-4fbd-8765-d7144d767d1c","Type":"ContainerStarted","Data":"3e924addd1bdbd6df1aa3c99e50270ea1d2e82c80e456e3e3463320aed7110f2"}
Oct 31 13:15:01.891421 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.890452    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" event={"ID":"f4b8769e-6fce-4fbd-8765-d7144d767d1c","Type":"ContainerStarted","Data":"3b55af145e59cd652196ff28c1d6479803889899c59f7be4ae5550ea78fe62f3"}
Oct 31 13:15:01.895778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.894083    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" event={"ID":"52986769-9ebe-449b-8e8b-1b1e82579282","Type":"ContainerStarted","Data":"8e720a916cc30a1aa0a3496c0ea946434d34860d84f8b6e41667b6898a834b68"}
Oct 31 13:15:01.895778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.894148    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" event={"ID":"52986769-9ebe-449b-8e8b-1b1e82579282","Type":"ContainerStarted","Data":"46d22a0ad43be3ab13d82057b3ba84c63a6e14fd8bfbb9fdb233d9ef34c8d4f4"}
Oct 31 13:15:01.899289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.898339    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" event={"ID":"9e1fa248-4cdb-4a4d-99e4-d9a259547d84","Type":"ContainerStarted","Data":"1604ef966160ce63ba63f9dbd338f2f9053913fee01302e2557b23395a5592e3"}
Oct 31 13:15:01.899289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.898398    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" event={"ID":"9e1fa248-4cdb-4a4d-99e4-d9a259547d84","Type":"ContainerStarted","Data":"9cb8a80d58b0646091f3e8a3a0188ef51a44d0edc39244b433519226e97835f6"}
Oct 31 13:15:01.911028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.910799    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" podStartSLOduration=1.910780629 podStartE2EDuration="1.910780629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:01.902113314 +0000 UTC m=+881.318055156" watchObservedRunningTime="2026-10-31 13:15:01.910780629 +0000 UTC m=+881.326722471"
Oct 31 13:15:01.911175 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.911051    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:01.927241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.927186    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" podStartSLOduration=1.927165996 podStartE2EDuration="1.927165996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:01.923225885 +0000 UTC m=+881.339167726" watchObservedRunningTime="2026-10-31 13:15:01.927165996 +0000 UTC m=+881.343107842"
Oct 31 13:15:01.954931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.954882    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" podStartSLOduration=1.9548673239999999 podStartE2EDuration="1.954867324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:01.950886147 +0000 UTC m=+881.366827988" watchObservedRunningTime="2026-10-31 13:15:01.954867324 +0000 UTC m=+881.370809154"
Oct 31 13:15:01.977968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:01.977904    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" podStartSLOduration=1.9778849219999999 podStartE2EDuration="1.977884922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:01.972780417 +0000 UTC m=+881.388722256" watchObservedRunningTime="2026-10-31 13:15:01.977884922 +0000 UTC m=+881.393826762"
Oct 31 13:15:02.213155 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:02.213124    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78ea089d_9578_4ea5_9361_0d0452da506d.slice/crio-ce256aa427a1b8c3d972b091a8430856bf6084e100c0a69be43ee1d1b3034632 WatchSource:0}: Error finding container ce256aa427a1b8c3d972b091a8430856bf6084e100c0a69be43ee1d1b3034632: Status 404 returned error can't find the container with id ce256aa427a1b8c3d972b091a8430856bf6084e100c0a69be43ee1d1b3034632
Oct 31 13:15:02.213293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.213236    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"]
Oct 31 13:15:02.907760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.907627    5787 generic.go:356] "Generic (PLEG): container finished" podID="52986769-9ebe-449b-8e8b-1b1e82579282" containerID="8e720a916cc30a1aa0a3496c0ea946434d34860d84f8b6e41667b6898a834b68" exitCode=60
Oct 31 13:15:02.907760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.907727    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" event={"ID":"52986769-9ebe-449b-8e8b-1b1e82579282","Type":"ContainerDied","Data":"8e720a916cc30a1aa0a3496c0ea946434d34860d84f8b6e41667b6898a834b68"}
Oct 31 13:15:02.908257 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.908135    5787 scope.go:122] "RemoveContainer" containerID="8e720a916cc30a1aa0a3496c0ea946434d34860d84f8b6e41667b6898a834b68"
Oct 31 13:15:02.909633 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.909607    5787 generic.go:356] "Generic (PLEG): container finished" podID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerID="1604ef966160ce63ba63f9dbd338f2f9053913fee01302e2557b23395a5592e3" exitCode=60
Oct 31 13:15:02.909811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.909688    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" event={"ID":"9e1fa248-4cdb-4a4d-99e4-d9a259547d84","Type":"ContainerDied","Data":"1604ef966160ce63ba63f9dbd338f2f9053913fee01302e2557b23395a5592e3"}
Oct 31 13:15:02.910083 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.910064    5787 scope.go:122] "RemoveContainer" containerID="1604ef966160ce63ba63f9dbd338f2f9053913fee01302e2557b23395a5592e3"
Oct 31 13:15:02.911983 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.911961    5787 generic.go:356] "Generic (PLEG): container finished" podID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerID="7c24bac43936dcb8a6d30ed26d6b433068ddfc78756a17d1ef2cd0e386afd1e1" exitCode=60
Oct 31 13:15:02.912100 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.912083    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" event={"ID":"3736d19a-1161-42bc-bf98-ec9d6408a563","Type":"ContainerDied","Data":"7c24bac43936dcb8a6d30ed26d6b433068ddfc78756a17d1ef2cd0e386afd1e1"}
Oct 31 13:15:02.912381 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.912360    5787 scope.go:122] "RemoveContainer" containerID="7c24bac43936dcb8a6d30ed26d6b433068ddfc78756a17d1ef2cd0e386afd1e1"
Oct 31 13:15:02.914095 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.914075    5787 generic.go:356] "Generic (PLEG): container finished" podID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerID="3e924addd1bdbd6df1aa3c99e50270ea1d2e82c80e456e3e3463320aed7110f2" exitCode=60
Oct 31 13:15:02.914176 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.914155    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" event={"ID":"f4b8769e-6fce-4fbd-8765-d7144d767d1c","Type":"ContainerDied","Data":"3e924addd1bdbd6df1aa3c99e50270ea1d2e82c80e456e3e3463320aed7110f2"}
Oct 31 13:15:02.914434 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.914417    5787 scope.go:122] "RemoveContainer" containerID="3e924addd1bdbd6df1aa3c99e50270ea1d2e82c80e456e3e3463320aed7110f2"
Oct 31 13:15:02.915945 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.915924    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj" event={"ID":"78ea089d-9578-4ea5-9361-0d0452da506d","Type":"ContainerStarted","Data":"4687dcdd0eef1ae80bbc8ab46afe7d1c2403c9db4a079916c186120949b20dca"}
Oct 31 13:15:02.916018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:02.915967    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj" event={"ID":"78ea089d-9578-4ea5-9361-0d0452da506d","Type":"ContainerStarted","Data":"ce256aa427a1b8c3d972b091a8430856bf6084e100c0a69be43ee1d1b3034632"}
Oct 31 13:15:03.059346 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:03.059281    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj" podStartSLOduration=2.059258437 podStartE2EDuration="2.059258437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:03.057113257 +0000 UTC m=+882.473055126" watchObservedRunningTime="2026-10-31 13:15:03.059258437 +0000 UTC m=+882.475200281"
Oct 31 13:15:03.984739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:03.981595    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" event={"ID":"3736d19a-1161-42bc-bf98-ec9d6408a563","Type":"ContainerStarted","Data":"da180e221cf9ec7d0b215b92908bdcaed2757f07ca158b7ec294dfbd9136590a"}
Oct 31 13:15:03.986742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:03.986294    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" event={"ID":"f4b8769e-6fce-4fbd-8765-d7144d767d1c","Type":"ContainerStarted","Data":"0ae4ea10cc549d5c33a22d32ff3be041e8fe22ba8ac99c41ad46bbf247e1fad3"}
Oct 31 13:15:03.991785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:03.989755    5787 generic.go:356] "Generic (PLEG): container finished" podID="78ea089d-9578-4ea5-9361-0d0452da506d" containerID="4687dcdd0eef1ae80bbc8ab46afe7d1c2403c9db4a079916c186120949b20dca" exitCode=0
Oct 31 13:15:03.991785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:03.989857    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj" event={"ID":"78ea089d-9578-4ea5-9361-0d0452da506d","Type":"ContainerDied","Data":"4687dcdd0eef1ae80bbc8ab46afe7d1c2403c9db4a079916c186120949b20dca"}
Oct 31 13:15:03.994756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:03.994345    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" event={"ID":"52986769-9ebe-449b-8e8b-1b1e82579282","Type":"ContainerStarted","Data":"3f5ab7cf0d4412e679ee0939e495f7ce2c3c4feb938156e52c299a23c1099e8d"}
Oct 31 13:15:03.997846 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:03.997813    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" event={"ID":"9e1fa248-4cdb-4a4d-99e4-d9a259547d84","Type":"ContainerStarted","Data":"10443eb208bf881d45a80eb43058e9449891683578d6bb6934cd7d4ed3fe92db"}
Oct 31 13:15:04.148916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.148882    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"]
Oct 31 13:15:04.153433 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.153410    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.180434 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.180389    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"]
Oct 31 13:15:04.238483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.238443    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7t2\" (UniqueName: \"kubernetes.io/projected/a1925a03-bd6a-437b-aa44-5f8c2741cd15-kube-api-access-wk7t2\") pod \"maas-api-revoke-sub-2dac9536-fq5d5\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.238640 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.238593    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dac9536-fq5d5\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.238890 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.238863    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dac9536-fq5d5\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.340619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.340582    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dac9536-fq5d5\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.340874 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.340788    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dac9536-fq5d5\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.340982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.340964    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7t2\" (UniqueName: \"kubernetes.io/projected/a1925a03-bd6a-437b-aa44-5f8c2741cd15-kube-api-access-wk7t2\") pod \"maas-api-revoke-sub-2dac9536-fq5d5\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.341506 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.341465    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dac9536-fq5d5\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.345062 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.345028    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dac9536-fq5d5\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.358354 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.358327    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7t2\" (UniqueName: \"kubernetes.io/projected/a1925a03-bd6a-437b-aa44-5f8c2741cd15-kube-api-access-wk7t2\") pod \"maas-api-revoke-sub-2dac9536-fq5d5\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.500839 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.500698    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:04.842063 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:04.842031    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1925a03_bd6a_437b_aa44_5f8c2741cd15.slice/crio-27d30f24e95819cdc57cdf5efdbc0f53ddecb7abcd40e0ff10e768c183e0b7af WatchSource:0}: Error finding container 27d30f24e95819cdc57cdf5efdbc0f53ddecb7abcd40e0ff10e768c183e0b7af: Status 404 returned error can't find the container with id 27d30f24e95819cdc57cdf5efdbc0f53ddecb7abcd40e0ff10e768c183e0b7af
Oct 31 13:15:04.845882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:04.845856    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"]
Oct 31 13:15:05.010852 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.010817    5787 generic.go:356] "Generic (PLEG): container finished" podID="52986769-9ebe-449b-8e8b-1b1e82579282" containerID="3f5ab7cf0d4412e679ee0939e495f7ce2c3c4feb938156e52c299a23c1099e8d" exitCode=60
Oct 31 13:15:05.011325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.010911    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" event={"ID":"52986769-9ebe-449b-8e8b-1b1e82579282","Type":"ContainerDied","Data":"3f5ab7cf0d4412e679ee0939e495f7ce2c3c4feb938156e52c299a23c1099e8d"}
Oct 31 13:15:05.011325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.010976    5787 scope.go:122] "RemoveContainer" containerID="8e720a916cc30a1aa0a3496c0ea946434d34860d84f8b6e41667b6898a834b68"
Oct 31 13:15:05.011456 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.011362    5787 scope.go:122] "RemoveContainer" containerID="3f5ab7cf0d4412e679ee0939e495f7ce2c3c4feb938156e52c299a23c1099e8d"
Oct 31 13:15:05.011891 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:05.011837    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l_odh-ai-gateway-infra(52986769-9ebe-449b-8e8b-1b1e82579282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" podUID="52986769-9ebe-449b-8e8b-1b1e82579282"
Oct 31 13:15:05.013526 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.013502    5787 generic.go:356] "Generic (PLEG): container finished" podID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerID="10443eb208bf881d45a80eb43058e9449891683578d6bb6934cd7d4ed3fe92db" exitCode=60
Oct 31 13:15:05.013648 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.013559    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" event={"ID":"9e1fa248-4cdb-4a4d-99e4-d9a259547d84","Type":"ContainerDied","Data":"10443eb208bf881d45a80eb43058e9449891683578d6bb6934cd7d4ed3fe92db"}
Oct 31 13:15:05.014031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.013988    5787 scope.go:122] "RemoveContainer" containerID="10443eb208bf881d45a80eb43058e9449891683578d6bb6934cd7d4ed3fe92db"
Oct 31 13:15:05.014428 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:05.014403    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw_odh-ai-gateway-infra(9e1fa248-4cdb-4a4d-99e4-d9a259547d84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84"
Oct 31 13:15:05.015755 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.015732    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" event={"ID":"3736d19a-1161-42bc-bf98-ec9d6408a563","Type":"ContainerDied","Data":"da180e221cf9ec7d0b215b92908bdcaed2757f07ca158b7ec294dfbd9136590a"}
Oct 31 13:15:05.015867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.015698    5787 generic.go:356] "Generic (PLEG): container finished" podID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerID="da180e221cf9ec7d0b215b92908bdcaed2757f07ca158b7ec294dfbd9136590a" exitCode=60
Oct 31 13:15:05.016129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.016111    5787 scope.go:122] "RemoveContainer" containerID="da180e221cf9ec7d0b215b92908bdcaed2757f07ca158b7ec294dfbd9136590a"
Oct 31 13:15:05.016551 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:05.016526    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd_odh-ai-gateway-infra(3736d19a-1161-42bc-bf98-ec9d6408a563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563"
Oct 31 13:15:05.018586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.018561    5787 generic.go:356] "Generic (PLEG): container finished" podID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerID="0ae4ea10cc549d5c33a22d32ff3be041e8fe22ba8ac99c41ad46bbf247e1fad3" exitCode=60
Oct 31 13:15:05.018683 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.018650    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" event={"ID":"f4b8769e-6fce-4fbd-8765-d7144d767d1c","Type":"ContainerDied","Data":"0ae4ea10cc549d5c33a22d32ff3be041e8fe22ba8ac99c41ad46bbf247e1fad3"}
Oct 31 13:15:05.018992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.018972    5787 scope.go:122] "RemoveContainer" containerID="0ae4ea10cc549d5c33a22d32ff3be041e8fe22ba8ac99c41ad46bbf247e1fad3"
Oct 31 13:15:05.019383 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:05.019358    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8_odh-ai-gateway-infra(f4b8769e-6fce-4fbd-8765-d7144d767d1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c"
Oct 31 13:15:05.020533 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.020508    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5" event={"ID":"a1925a03-bd6a-437b-aa44-5f8c2741cd15","Type":"ContainerStarted","Data":"27d30f24e95819cdc57cdf5efdbc0f53ddecb7abcd40e0ff10e768c183e0b7af"}
Oct 31 13:15:05.029102 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.029068    5787 scope.go:122] "RemoveContainer" containerID="1604ef966160ce63ba63f9dbd338f2f9053913fee01302e2557b23395a5592e3"
Oct 31 13:15:05.067182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.067050    5787 scope.go:122] "RemoveContainer" containerID="7c24bac43936dcb8a6d30ed26d6b433068ddfc78756a17d1ef2cd0e386afd1e1"
Oct 31 13:15:05.086254 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.086226    5787 scope.go:122] "RemoveContainer" containerID="3e924addd1bdbd6df1aa3c99e50270ea1d2e82c80e456e3e3463320aed7110f2"
Oct 31 13:15:05.208811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.208780    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:05.360010 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.359913    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ea089d-9578-4ea5-9361-0d0452da506d-kube-api-access-d5p97\" (UniqueName: \"kubernetes.io/projected/78ea089d-9578-4ea5-9361-0d0452da506d-kube-api-access-d5p97\") pod \"78ea089d-9578-4ea5-9361-0d0452da506d\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") "
Oct 31 13:15:05.360185 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.360055    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-xks-ca\") pod \"78ea089d-9578-4ea5-9361-0d0452da506d\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") "
Oct 31 13:15:05.360185 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.360124    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-openshift-ca\") pod \"78ea089d-9578-4ea5-9361-0d0452da506d\" (UID: \"78ea089d-9578-4ea5-9361-0d0452da506d\") "
Oct 31 13:15:05.360527 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.360501    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-openshift-ca" pod "78ea089d-9578-4ea5-9361-0d0452da506d" (UID: "78ea089d-9578-4ea5-9361-0d0452da506d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:05.360881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.360853    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:05.362466 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.362440    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ea089d-9578-4ea5-9361-0d0452da506d-kube-api-access-d5p97" pod "78ea089d-9578-4ea5-9361-0d0452da506d" (UID: "78ea089d-9578-4ea5-9361-0d0452da506d"). InnerVolumeSpecName "kube-api-access-d5p97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:05.362536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.362492    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-xks-ca" pod "78ea089d-9578-4ea5-9361-0d0452da506d" (UID: "78ea089d-9578-4ea5-9361-0d0452da506d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:05.463058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.463020    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5p97\" (UniqueName: \"kubernetes.io/projected/78ea089d-9578-4ea5-9361-0d0452da506d-kube-api-access-d5p97\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:05.463058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:05.463055    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78ea089d-9578-4ea5-9361-0d0452da506d-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:06.030116 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.030087    5787 scope.go:122] "RemoveContainer" containerID="10443eb208bf881d45a80eb43058e9449891683578d6bb6934cd7d4ed3fe92db"
Oct 31 13:15:06.030530 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:06.030491    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw_odh-ai-gateway-infra(9e1fa248-4cdb-4a4d-99e4-d9a259547d84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84"
Oct 31 13:15:06.031974 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.031952    5787 scope.go:122] "RemoveContainer" containerID="da180e221cf9ec7d0b215b92908bdcaed2757f07ca158b7ec294dfbd9136590a"
Oct 31 13:15:06.032309 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:06.032288    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd_odh-ai-gateway-infra(3736d19a-1161-42bc-bf98-ec9d6408a563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563"
Oct 31 13:15:06.033821 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.033796    5787 scope.go:122] "RemoveContainer" containerID="0ae4ea10cc549d5c33a22d32ff3be041e8fe22ba8ac99c41ad46bbf247e1fad3"
Oct 31 13:15:06.034184 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:06.034150    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8_odh-ai-gateway-infra(f4b8769e-6fce-4fbd-8765-d7144d767d1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c"
Oct 31 13:15:06.035260 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.035234    5787 generic.go:356] "Generic (PLEG): container finished" podID="a1925a03-bd6a-437b-aa44-5f8c2741cd15" containerID="c45b09a8fc885dbec0f1ba8461671cc80d9cbe52673f8e65e0b5317ede5cdcd6" exitCode=0
Oct 31 13:15:06.035365 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.035273    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5" event={"ID":"a1925a03-bd6a-437b-aa44-5f8c2741cd15","Type":"ContainerDied","Data":"c45b09a8fc885dbec0f1ba8461671cc80d9cbe52673f8e65e0b5317ede5cdcd6"}
Oct 31 13:15:06.036882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.036865    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"
Oct 31 13:15:06.036996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.036883    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj" event={"ID":"78ea089d-9578-4ea5-9361-0d0452da506d","Type":"ContainerDied","Data":"ce256aa427a1b8c3d972b091a8430856bf6084e100c0a69be43ee1d1b3034632"}
Oct 31 13:15:06.036996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.036921    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce256aa427a1b8c3d972b091a8430856bf6084e100c0a69be43ee1d1b3034632"
Oct 31 13:15:06.038833 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.038814    5787 scope.go:122] "RemoveContainer" containerID="3f5ab7cf0d4412e679ee0939e495f7ce2c3c4feb938156e52c299a23c1099e8d"
Oct 31 13:15:06.039067 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:06.039048    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l_odh-ai-gateway-infra(52986769-9ebe-449b-8e8b-1b1e82579282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" podUID="52986769-9ebe-449b-8e8b-1b1e82579282"
Oct 31 13:15:06.071195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.071159    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"]
Oct 31 13:15:06.072101 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.072075    5787 cpu_manager.go:414] "containerMap: removing container" podUID="78ea089d-9578-4ea5-9361-0d0452da506d" containerName="revoke-keys"
Oct 31 13:15:06.072101 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.072101    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ea089d-9578-4ea5-9361-0d0452da506d" containerName="revoke-keys"
Oct 31 13:15:06.072400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.072384    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ea089d-9578-4ea5-9361-0d0452da506d" containerName="revoke-keys"
Oct 31 13:15:06.077129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.077108    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.091496 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.091462    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"]
Oct 31 13:15:06.174953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.174922    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.175216 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.175184    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.175722 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.175678    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnk4c\" (UniqueName: \"kubernetes.io/projected/8b35b374-4ada-444e-8947-c1abf0bf5fe7-kube-api-access-qnk4c\") pod \"maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.221266 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.221183    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"]
Oct 31 13:15:06.232129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.232097    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"]
Oct 31 13:15:06.239052 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.239015    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"]
Oct 31 13:15:06.277158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.277128    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnk4c\" (UniqueName: \"kubernetes.io/projected/8b35b374-4ada-444e-8947-c1abf0bf5fe7-kube-api-access-qnk4c\") pod \"maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.277340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.277176    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.277340 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.277308    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.278041 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.277970    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.279862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.279825    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.291013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.290981    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnk4c\" (UniqueName: \"kubernetes.io/projected/8b35b374-4ada-444e-8947-c1abf0bf5fe7-kube-api-access-qnk4c\") pod \"maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.393103 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.393070    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:06.680754 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:06.680718    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b35b374_4ada_444e_8947_c1abf0bf5fe7.slice/crio-ed73272467972e9164d2d1a6b282e3fcb62251d7bf3cc01f786cea0a4789d5da WatchSource:0}: Error finding container ed73272467972e9164d2d1a6b282e3fcb62251d7bf3cc01f786cea0a4789d5da: Status 404 returned error can't find the container with id ed73272467972e9164d2d1a6b282e3fcb62251d7bf3cc01f786cea0a4789d5da
Oct 31 13:15:06.681253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:06.681233    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"]
Oct 31 13:15:07.049557 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.049527    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm" event={"ID":"8b35b374-4ada-444e-8947-c1abf0bf5fe7","Type":"ContainerStarted","Data":"ed73272467972e9164d2d1a6b282e3fcb62251d7bf3cc01f786cea0a4789d5da"}
Oct 31 13:15:07.240063 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.240034    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:07.402250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.402158    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-openshift-ca\") pod \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") "
Oct 31 13:15:07.402250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.402194    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-xks-ca\") pod \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") "
Oct 31 13:15:07.402491 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.402326    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1925a03-bd6a-437b-aa44-5f8c2741cd15-kube-api-access-wk7t2\" (UniqueName: \"kubernetes.io/projected/a1925a03-bd6a-437b-aa44-5f8c2741cd15-kube-api-access-wk7t2\") pod \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\" (UID: \"a1925a03-bd6a-437b-aa44-5f8c2741cd15\") "
Oct 31 13:15:07.402559 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.402532    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-openshift-ca" pod "a1925a03-bd6a-437b-aa44-5f8c2741cd15" (UID: "a1925a03-bd6a-437b-aa44-5f8c2741cd15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:07.402967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.402945    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:07.404834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.404806    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-xks-ca" pod "a1925a03-bd6a-437b-aa44-5f8c2741cd15" (UID: "a1925a03-bd6a-437b-aa44-5f8c2741cd15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:07.404965 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.404874    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1925a03-bd6a-437b-aa44-5f8c2741cd15-kube-api-access-wk7t2" pod "a1925a03-bd6a-437b-aa44-5f8c2741cd15" (UID: "a1925a03-bd6a-437b-aa44-5f8c2741cd15"). InnerVolumeSpecName "kube-api-access-wk7t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:07.504561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.504527    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1925a03-bd6a-437b-aa44-5f8c2741cd15-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:07.504561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.504559    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wk7t2\" (UniqueName: \"kubernetes.io/projected/a1925a03-bd6a-437b-aa44-5f8c2741cd15-kube-api-access-wk7t2\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:07.691547 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.691509    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"]
Oct 31 13:15:07.692224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.692206    5787 cpu_manager.go:414] "containerMap: removing container" podUID="a1925a03-bd6a-437b-aa44-5f8c2741cd15" containerName="revoke-keys"
Oct 31 13:15:07.692224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.692224    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1925a03-bd6a-437b-aa44-5f8c2741cd15" containerName="revoke-keys"
Oct 31 13:15:07.692370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.692361    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1925a03-bd6a-437b-aa44-5f8c2741cd15" containerName="revoke-keys"
Oct 31 13:15:07.696100 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.696084    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:07.706383 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.706352    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"]
Oct 31 13:15:07.808982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.808949    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f77003-5w5mr\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:07.809154 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.809028    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g5fv\" (UniqueName: \"kubernetes.io/projected/371beb92-8f45-480b-bd39-6bf4d1d9d987-kube-api-access-9g5fv\") pod \"maas-api-revoke-sub-e6f77003-5w5mr\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:07.809219 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.809201    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f77003-5w5mr\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:07.911285 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.911251    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f77003-5w5mr\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:07.911454 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.911408    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f77003-5w5mr\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:07.911508 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.911474    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g5fv\" (UniqueName: \"kubernetes.io/projected/371beb92-8f45-480b-bd39-6bf4d1d9d987-kube-api-access-9g5fv\") pod \"maas-api-revoke-sub-e6f77003-5w5mr\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:07.912228 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.912200    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6f77003-5w5mr\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:07.914035 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.914012    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6f77003-5w5mr\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:07.925331 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:07.925305    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g5fv\" (UniqueName: \"kubernetes.io/projected/371beb92-8f45-480b-bd39-6bf4d1d9d987-kube-api-access-9g5fv\") pod \"maas-api-revoke-sub-e6f77003-5w5mr\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:08.013513 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:08.013478    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:08.060657 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:08.060625    5787 generic.go:356] "Generic (PLEG): container finished" podID="8b35b374-4ada-444e-8947-c1abf0bf5fe7" containerID="2a798ebd143819ed2526e276e4fa130076ac50d932c97636990ce98a2568b3aa" exitCode=0
Oct 31 13:15:08.061131 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:08.060720    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm" event={"ID":"8b35b374-4ada-444e-8947-c1abf0bf5fe7","Type":"ContainerDied","Data":"2a798ebd143819ed2526e276e4fa130076ac50d932c97636990ce98a2568b3aa"}
Oct 31 13:15:08.063253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:08.062572    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5" event={"ID":"a1925a03-bd6a-437b-aa44-5f8c2741cd15","Type":"ContainerDied","Data":"27d30f24e95819cdc57cdf5efdbc0f53ddecb7abcd40e0ff10e768c183e0b7af"}
Oct 31 13:15:08.063253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:08.062621    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27d30f24e95819cdc57cdf5efdbc0f53ddecb7abcd40e0ff10e768c183e0b7af"
Oct 31 13:15:08.063253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:08.062579    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"
Oct 31 13:15:08.359243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:08.356847    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"]
Oct 31 13:15:09.073131 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.073098    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr" event={"ID":"371beb92-8f45-480b-bd39-6bf4d1d9d987","Type":"ContainerStarted","Data":"ab64b75a219a1932f280078f8f83f3cdaf9542789461ba0f6b3d16f90cb1f515"}
Oct 31 13:15:09.073606 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.073160    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr" event={"ID":"371beb92-8f45-480b-bd39-6bf4d1d9d987","Type":"ContainerStarted","Data":"875cc1a86cf4ebbd394d377d0b08210876da0c456d16e1cce3a43dfaae9bc9b4"}
Oct 31 13:15:09.110591 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.110532    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr" podStartSLOduration=2.110516819 podStartE2EDuration="2.110516819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:09.106871684 +0000 UTC m=+888.522813524" watchObservedRunningTime="2026-10-31 13:15:09.110516819 +0000 UTC m=+888.526458686"
Oct 31 13:15:09.363771 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.362935    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:09.370933 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.370899    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-openshift-ca\") pod \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") "
Oct 31 13:15:09.371110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.371093    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b35b374-4ada-444e-8947-c1abf0bf5fe7-kube-api-access-qnk4c\" (UniqueName: \"kubernetes.io/projected/8b35b374-4ada-444e-8947-c1abf0bf5fe7-kube-api-access-qnk4c\") pod \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") "
Oct 31 13:15:09.371201 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.371184    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-xks-ca\") pod \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\" (UID: \"8b35b374-4ada-444e-8947-c1abf0bf5fe7\") "
Oct 31 13:15:09.371331 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.371306    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-openshift-ca" pod "8b35b374-4ada-444e-8947-c1abf0bf5fe7" (UID: "8b35b374-4ada-444e-8947-c1abf0bf5fe7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:09.372034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.372012    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:09.373629 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.373605    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-xks-ca" pod "8b35b374-4ada-444e-8947-c1abf0bf5fe7" (UID: "8b35b374-4ada-444e-8947-c1abf0bf5fe7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:09.373927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.373906    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b35b374-4ada-444e-8947-c1abf0bf5fe7-kube-api-access-qnk4c" pod "8b35b374-4ada-444e-8947-c1abf0bf5fe7" (UID: "8b35b374-4ada-444e-8947-c1abf0bf5fe7"). InnerVolumeSpecName "kube-api-access-qnk4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:09.473509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.473475    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnk4c\" (UniqueName: \"kubernetes.io/projected/8b35b374-4ada-444e-8947-c1abf0bf5fe7-kube-api-access-qnk4c\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:09.473509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.473508    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b35b374-4ada-444e-8947-c1abf0bf5fe7-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:09.604840 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.604795    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"]
Oct 31 13:15:09.605544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.605523    5787 cpu_manager.go:414] "containerMap: removing container" podUID="8b35b374-4ada-444e-8947-c1abf0bf5fe7" containerName="revoke-keys"
Oct 31 13:15:09.605644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.605545    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b35b374-4ada-444e-8947-c1abf0bf5fe7" containerName="revoke-keys"
Oct 31 13:15:09.605818 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.605801    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b35b374-4ada-444e-8947-c1abf0bf5fe7" containerName="revoke-keys"
Oct 31 13:15:09.609556 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.609535    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.618760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.618630    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"]
Oct 31 13:15:09.676751 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.676693    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fdd5189-vbqv5\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.677009 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.676983    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzt5d\" (UniqueName: \"kubernetes.io/projected/65b95431-cb3c-458b-b585-4afa8c808590-kube-api-access-mzt5d\") pod \"maas-api-revoke-sub-8fdd5189-vbqv5\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.677062 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.677025    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fdd5189-vbqv5\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.778293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.778254    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzt5d\" (UniqueName: \"kubernetes.io/projected/65b95431-cb3c-458b-b585-4afa8c808590-kube-api-access-mzt5d\") pod \"maas-api-revoke-sub-8fdd5189-vbqv5\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.778293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.778290    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fdd5189-vbqv5\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.778518 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.778355    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fdd5189-vbqv5\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.779106 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.779083    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fdd5189-vbqv5\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.780909 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.780883    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fdd5189-vbqv5\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.791597 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.791570    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzt5d\" (UniqueName: \"kubernetes.io/projected/65b95431-cb3c-458b-b585-4afa8c808590-kube-api-access-mzt5d\") pod \"maas-api-revoke-sub-8fdd5189-vbqv5\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:09.930144 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:09.930057    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:10.177078 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:10.175850    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm" event={"ID":"8b35b374-4ada-444e-8947-c1abf0bf5fe7","Type":"ContainerDied","Data":"ed73272467972e9164d2d1a6b282e3fcb62251d7bf3cc01f786cea0a4789d5da"}
Oct 31 13:15:10.177078 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:10.175913    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed73272467972e9164d2d1a6b282e3fcb62251d7bf3cc01f786cea0a4789d5da"
Oct 31 13:15:10.177078 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:10.175998    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"
Oct 31 13:15:10.199544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:10.199513    5787 generic.go:356] "Generic (PLEG): container finished" podID="371beb92-8f45-480b-bd39-6bf4d1d9d987" containerID="ab64b75a219a1932f280078f8f83f3cdaf9542789461ba0f6b3d16f90cb1f515" exitCode=0
Oct 31 13:15:10.199742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:10.199677    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr" event={"ID":"371beb92-8f45-480b-bd39-6bf4d1d9d987","Type":"ContainerDied","Data":"ab64b75a219a1932f280078f8f83f3cdaf9542789461ba0f6b3d16f90cb1f515"}
Oct 31 13:15:10.240104 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:10.233079    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"]
Oct 31 13:15:10.240104 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:10.240050    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b95431_cb3c_458b_b585_4afa8c808590.slice/crio-bf7dc451bc3e1e00f6b5d42f88c96b47bd9c3277a3f70d189fb4db2c4e8de21c WatchSource:0}: Error finding container bf7dc451bc3e1e00f6b5d42f88c96b47bd9c3277a3f70d189fb4db2c4e8de21c: Status 404 returned error can't find the container with id bf7dc451bc3e1e00f6b5d42f88c96b47bd9c3277a3f70d189fb4db2c4e8de21c
Oct 31 13:15:10.920814 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:10.916899    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"]
Oct 31 13:15:10.937551 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:10.934561    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"]
Oct 31 13:15:11.210012 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.209978    5787 generic.go:356] "Generic (PLEG): container finished" podID="65b95431-cb3c-458b-b585-4afa8c808590" containerID="6bdead7a24befaa37918e5b6014dbf8c972e0081034a0bc0974b304346880cf0" exitCode=0
Oct 31 13:15:11.210449 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.210120    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5" event={"ID":"65b95431-cb3c-458b-b585-4afa8c808590","Type":"ContainerDied","Data":"6bdead7a24befaa37918e5b6014dbf8c972e0081034a0bc0974b304346880cf0"}
Oct 31 13:15:11.210449 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.210250    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5" event={"ID":"65b95431-cb3c-458b-b585-4afa8c808590","Type":"ContainerStarted","Data":"bf7dc451bc3e1e00f6b5d42f88c96b47bd9c3277a3f70d189fb4db2c4e8de21c"}
Oct 31 13:15:11.375691 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.375667    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:11.434017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.433982    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371beb92-8f45-480b-bd39-6bf4d1d9d987-kube-api-access-9g5fv\" (UniqueName: \"kubernetes.io/projected/371beb92-8f45-480b-bd39-6bf4d1d9d987-kube-api-access-9g5fv\") pod \"371beb92-8f45-480b-bd39-6bf4d1d9d987\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") "
Oct 31 13:15:11.434272 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.434253    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-openshift-ca\") pod \"371beb92-8f45-480b-bd39-6bf4d1d9d987\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") "
Oct 31 13:15:11.434479 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.434452    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-xks-ca\") pod \"371beb92-8f45-480b-bd39-6bf4d1d9d987\" (UID: \"371beb92-8f45-480b-bd39-6bf4d1d9d987\") "
Oct 31 13:15:11.434694 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.434653    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-openshift-ca" pod "371beb92-8f45-480b-bd39-6bf4d1d9d987" (UID: "371beb92-8f45-480b-bd39-6bf4d1d9d987"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:11.435095 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.435073    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:11.436493 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.436471    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371beb92-8f45-480b-bd39-6bf4d1d9d987-kube-api-access-9g5fv" pod "371beb92-8f45-480b-bd39-6bf4d1d9d987" (UID: "371beb92-8f45-480b-bd39-6bf4d1d9d987"). InnerVolumeSpecName "kube-api-access-9g5fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:11.437132 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.437113    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-xks-ca" pod "371beb92-8f45-480b-bd39-6bf4d1d9d987" (UID: "371beb92-8f45-480b-bd39-6bf4d1d9d987"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:11.536223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.536140    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371beb92-8f45-480b-bd39-6bf4d1d9d987-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:11.536223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:11.536172    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g5fv\" (UniqueName: \"kubernetes.io/projected/371beb92-8f45-480b-bd39-6bf4d1d9d987-kube-api-access-9g5fv\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:12.339731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.333821    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"
Oct 31 13:15:12.339731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.338493    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr" event={"ID":"371beb92-8f45-480b-bd39-6bf4d1d9d987","Type":"ContainerDied","Data":"875cc1a86cf4ebbd394d377d0b08210876da0c456d16e1cce3a43dfaae9bc9b4"}
Oct 31 13:15:12.339731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.338558    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="875cc1a86cf4ebbd394d377d0b08210876da0c456d16e1cce3a43dfaae9bc9b4"
Oct 31 13:15:12.588811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.584042    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"]
Oct 31 13:15:12.588811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.585195    5787 cpu_manager.go:414] "containerMap: removing container" podUID="371beb92-8f45-480b-bd39-6bf4d1d9d987" containerName="revoke-keys"
Oct 31 13:15:12.588811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.585524    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371beb92-8f45-480b-bd39-6bf4d1d9d987" containerName="revoke-keys"
Oct 31 13:15:12.588811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.585823    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="371beb92-8f45-480b-bd39-6bf4d1d9d987" containerName="revoke-keys"
Oct 31 13:15:12.598248 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.594661    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.598248 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.597693    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"]
Oct 31 13:15:12.649344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.649314    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:12.694598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.694556    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-xks-ca\") pod \"65b95431-cb3c-458b-b585-4afa8c808590\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") "
Oct 31 13:15:12.694810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.694612    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-openshift-ca\") pod \"65b95431-cb3c-458b-b585-4afa8c808590\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") "
Oct 31 13:15:12.694810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.694663    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b95431-cb3c-458b-b585-4afa8c808590-kube-api-access-mzt5d\" (UniqueName: \"kubernetes.io/projected/65b95431-cb3c-458b-b585-4afa8c808590-kube-api-access-mzt5d\") pod \"65b95431-cb3c-458b-b585-4afa8c808590\" (UID: \"65b95431-cb3c-458b-b585-4afa8c808590\") "
Oct 31 13:15:12.695054 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.695028    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-openshift-ca" pod "65b95431-cb3c-458b-b585-4afa8c808590" (UID: "65b95431-cb3c-458b-b585-4afa8c808590"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:12.695313 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.695262    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfwj\" (UniqueName: \"kubernetes.io/projected/8eb9a0f4-b481-496b-b905-e9667799c175-kube-api-access-8zfwj\") pod \"maas-api-revoke-sub-db78c405-nwdvt\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.695866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.695842    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db78c405-nwdvt\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.696008 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.695987    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db78c405-nwdvt\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.696212 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.696189    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:12.697275 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.697250    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-xks-ca" pod "65b95431-cb3c-458b-b585-4afa8c808590" (UID: "65b95431-cb3c-458b-b585-4afa8c808590"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:12.697376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.697285    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b95431-cb3c-458b-b585-4afa8c808590-kube-api-access-mzt5d" pod "65b95431-cb3c-458b-b585-4afa8c808590" (UID: "65b95431-cb3c-458b-b585-4afa8c808590"). InnerVolumeSpecName "kube-api-access-mzt5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:12.797555 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.797518    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db78c405-nwdvt\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.797792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.797581    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db78c405-nwdvt\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.797792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.797641    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfwj\" (UniqueName: \"kubernetes.io/projected/8eb9a0f4-b481-496b-b905-e9667799c175-kube-api-access-8zfwj\") pod \"maas-api-revoke-sub-db78c405-nwdvt\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.797932 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.797831    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65b95431-cb3c-458b-b585-4afa8c808590-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:12.797932 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.797843    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzt5d\" (UniqueName: \"kubernetes.io/projected/65b95431-cb3c-458b-b585-4afa8c808590-kube-api-access-mzt5d\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:12.798362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.798333    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db78c405-nwdvt\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.800295 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.800267    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db78c405-nwdvt\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.823198 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.823164    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfwj\" (UniqueName: \"kubernetes.io/projected/8eb9a0f4-b481-496b-b905-e9667799c175-kube-api-access-8zfwj\") pod \"maas-api-revoke-sub-db78c405-nwdvt\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:12.927468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:12.927381    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:13.197450 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.197420    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"]
Oct 31 13:15:13.198134 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.198117    5787 cpu_manager.go:414] "containerMap: removing container" podUID="65b95431-cb3c-458b-b585-4afa8c808590" containerName="revoke-keys"
Oct 31 13:15:13.198134 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.198134    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b95431-cb3c-458b-b585-4afa8c808590" containerName="revoke-keys"
Oct 31 13:15:13.198265 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.198255    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b95431-cb3c-458b-b585-4afa8c808590" containerName="revoke-keys"
Oct 31 13:15:13.204847 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.204820    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.207006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.206977    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"]
Oct 31 13:15:13.226736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.224687    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"]
Oct 31 13:15:13.310509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.310464    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392c74b1-chkf9\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.310698 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.310612    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392c74b1-chkf9\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.310898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.310880    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvm9d\" (UniqueName: \"kubernetes.io/projected/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-kube-api-access-xvm9d\") pod \"maas-api-revoke-sub-392c74b1-chkf9\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.344587 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.344550    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt" event={"ID":"8eb9a0f4-b481-496b-b905-e9667799c175","Type":"ContainerStarted","Data":"b9f30ca78e3a66f1bfa71ce984fa916066b34f6222e8322944d22ce0b3a51d36"}
Oct 31 13:15:13.346363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.346335    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5" event={"ID":"65b95431-cb3c-458b-b585-4afa8c808590","Type":"ContainerDied","Data":"bf7dc451bc3e1e00f6b5d42f88c96b47bd9c3277a3f70d189fb4db2c4e8de21c"}
Oct 31 13:15:13.346487 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.346383    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf7dc451bc3e1e00f6b5d42f88c96b47bd9c3277a3f70d189fb4db2c4e8de21c"
Oct 31 13:15:13.346487 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.346347    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"
Oct 31 13:15:13.412579 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.412542    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvm9d\" (UniqueName: \"kubernetes.io/projected/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-kube-api-access-xvm9d\") pod \"maas-api-revoke-sub-392c74b1-chkf9\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.412783 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.412638    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392c74b1-chkf9\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.412783 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.412699    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392c74b1-chkf9\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.413466 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.413447    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-392c74b1-chkf9\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.415343 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.415319    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-392c74b1-chkf9\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.426616 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.426588    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvm9d\" (UniqueName: \"kubernetes.io/projected/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-kube-api-access-xvm9d\") pod \"maas-api-revoke-sub-392c74b1-chkf9\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.533940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.533861    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:13.828017 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:13.827978    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1d0db60_46fc_4929_bcae_f0bb9f2e587b.slice/crio-76a9ab566c8d07383e87631566db16b5a6128c029594555822e8bbe9fecfbcd5 WatchSource:0}: Error finding container 76a9ab566c8d07383e87631566db16b5a6128c029594555822e8bbe9fecfbcd5: Status 404 returned error can't find the container with id 76a9ab566c8d07383e87631566db16b5a6128c029594555822e8bbe9fecfbcd5
Oct 31 13:15:13.828597 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:13.828569    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"]
Oct 31 13:15:14.357481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:14.357447    5787 generic.go:356] "Generic (PLEG): container finished" podID="8eb9a0f4-b481-496b-b905-e9667799c175" containerID="8a436775f6751def3060dde796d2b2d051e81985b40e31dbd22788a3719e1ab7" exitCode=0
Oct 31 13:15:14.357923 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:14.357488    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt" event={"ID":"8eb9a0f4-b481-496b-b905-e9667799c175","Type":"ContainerDied","Data":"8a436775f6751def3060dde796d2b2d051e81985b40e31dbd22788a3719e1ab7"}
Oct 31 13:15:14.359506 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:14.359475    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9" event={"ID":"e1d0db60-46fc-4929-bcae-f0bb9f2e587b","Type":"ContainerStarted","Data":"97d02905484e588cbf593ee1477ce42a60cdcefabd4c88e3de6e109f4978ece7"}
Oct 31 13:15:14.359650 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:14.359525    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9" event={"ID":"e1d0db60-46fc-4929-bcae-f0bb9f2e587b","Type":"ContainerStarted","Data":"76a9ab566c8d07383e87631566db16b5a6128c029594555822e8bbe9fecfbcd5"}
Oct 31 13:15:14.399837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:14.399779    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9" podStartSLOduration=1.399766103 podStartE2EDuration="1.399766103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:14.398363146 +0000 UTC m=+893.814304987" watchObservedRunningTime="2026-10-31 13:15:14.399766103 +0000 UTC m=+893.815707943"
Oct 31 13:15:14.699654 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:14.699619    5787 scope.go:122] "RemoveContainer" containerID="0ae4ea10cc549d5c33a22d32ff3be041e8fe22ba8ac99c41ad46bbf247e1fad3"
Oct 31 13:15:15.371613 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.371525    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" event={"ID":"f4b8769e-6fce-4fbd-8765-d7144d767d1c","Type":"ContainerStarted","Data":"48d602964cc39b0cebc63fb610ca4d7825c25929bd72c1ab5bddb8fe852e9022"}
Oct 31 13:15:15.373246 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.373222    5787 generic.go:356] "Generic (PLEG): container finished" podID="e1d0db60-46fc-4929-bcae-f0bb9f2e587b" containerID="97d02905484e588cbf593ee1477ce42a60cdcefabd4c88e3de6e109f4978ece7" exitCode=0
Oct 31 13:15:15.373361 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.373298    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9" event={"ID":"e1d0db60-46fc-4929-bcae-f0bb9f2e587b","Type":"ContainerDied","Data":"97d02905484e588cbf593ee1477ce42a60cdcefabd4c88e3de6e109f4978ece7"}
Oct 31 13:15:15.553624 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.553593    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:15.657534 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.657496    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-openshift-ca\") pod \"8eb9a0f4-b481-496b-b905-e9667799c175\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") "
Oct 31 13:15:15.657779 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.657762    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-xks-ca\") pod \"8eb9a0f4-b481-496b-b905-e9667799c175\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") "
Oct 31 13:15:15.657922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.657895    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-openshift-ca" pod "8eb9a0f4-b481-496b-b905-e9667799c175" (UID: "8eb9a0f4-b481-496b-b905-e9667799c175"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:15.657993 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.657921    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb9a0f4-b481-496b-b905-e9667799c175-kube-api-access-8zfwj\" (UniqueName: \"kubernetes.io/projected/8eb9a0f4-b481-496b-b905-e9667799c175-kube-api-access-8zfwj\") pod \"8eb9a0f4-b481-496b-b905-e9667799c175\" (UID: \"8eb9a0f4-b481-496b-b905-e9667799c175\") "
Oct 31 13:15:15.658459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.658437    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:15.660053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.660030    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-xks-ca" pod "8eb9a0f4-b481-496b-b905-e9667799c175" (UID: "8eb9a0f4-b481-496b-b905-e9667799c175"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:15.660232 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.660215    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb9a0f4-b481-496b-b905-e9667799c175-kube-api-access-8zfwj" pod "8eb9a0f4-b481-496b-b905-e9667799c175" (UID: "8eb9a0f4-b481-496b-b905-e9667799c175"). InnerVolumeSpecName "kube-api-access-8zfwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:15.698985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.698955    5787 scope.go:122] "RemoveContainer" containerID="da180e221cf9ec7d0b215b92908bdcaed2757f07ca158b7ec294dfbd9136590a"
Oct 31 13:15:15.760625 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.760595    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb9a0f4-b481-496b-b905-e9667799c175-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:15.760625 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:15.760627    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zfwj\" (UniqueName: \"kubernetes.io/projected/8eb9a0f4-b481-496b-b905-e9667799c175-kube-api-access-8zfwj\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:16.383509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.383415    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"
Oct 31 13:15:16.383987 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.383411    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt" event={"ID":"8eb9a0f4-b481-496b-b905-e9667799c175","Type":"ContainerDied","Data":"b9f30ca78e3a66f1bfa71ce984fa916066b34f6222e8322944d22ce0b3a51d36"}
Oct 31 13:15:16.383987 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.383559    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9f30ca78e3a66f1bfa71ce984fa916066b34f6222e8322944d22ce0b3a51d36"
Oct 31 13:15:16.385733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.385677    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" event={"ID":"3736d19a-1161-42bc-bf98-ec9d6408a563","Type":"ContainerStarted","Data":"3dbb6584d4489f31a4bff887a1e76c9829406f5c61aeb47d0a088bea33a39100"}
Oct 31 13:15:16.388913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.388890    5787 generic.go:356] "Generic (PLEG): container finished" podID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerID="48d602964cc39b0cebc63fb610ca4d7825c25929bd72c1ab5bddb8fe852e9022" exitCode=60
Oct 31 13:15:16.389070 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.388965    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" event={"ID":"f4b8769e-6fce-4fbd-8765-d7144d767d1c","Type":"ContainerDied","Data":"48d602964cc39b0cebc63fb610ca4d7825c25929bd72c1ab5bddb8fe852e9022"}
Oct 31 13:15:16.389070 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.389019    5787 scope.go:122] "RemoveContainer" containerID="0ae4ea10cc549d5c33a22d32ff3be041e8fe22ba8ac99c41ad46bbf247e1fad3"
Oct 31 13:15:16.390757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.389583    5787 scope.go:122] "RemoveContainer" containerID="48d602964cc39b0cebc63fb610ca4d7825c25929bd72c1ab5bddb8fe852e9022"
Oct 31 13:15:16.390757 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:16.390026    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8_odh-ai-gateway-infra(f4b8769e-6fce-4fbd-8765-d7144d767d1c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c"
Oct 31 13:15:16.421104 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.421061    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"]
Oct 31 13:15:16.900249 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:16.899842    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:17.018633 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.018591    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-xks-ca\") pod \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") "
Oct 31 13:15:17.018891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.018669    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-kube-api-access-xvm9d\" (UniqueName: \"kubernetes.io/projected/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-kube-api-access-xvm9d\") pod \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") "
Oct 31 13:15:17.018891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.018764    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-openshift-ca\") pod \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\" (UID: \"e1d0db60-46fc-4929-bcae-f0bb9f2e587b\") "
Oct 31 13:15:17.020195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.020087    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-openshift-ca" pod "e1d0db60-46fc-4929-bcae-f0bb9f2e587b" (UID: "e1d0db60-46fc-4929-bcae-f0bb9f2e587b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:17.022693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.022656    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-xks-ca" pod "e1d0db60-46fc-4929-bcae-f0bb9f2e587b" (UID: "e1d0db60-46fc-4929-bcae-f0bb9f2e587b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:17.023481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.023453    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-kube-api-access-xvm9d" pod "e1d0db60-46fc-4929-bcae-f0bb9f2e587b" (UID: "e1d0db60-46fc-4929-bcae-f0bb9f2e587b"). InnerVolumeSpecName "kube-api-access-xvm9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:17.121092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.121007    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvm9d\" (UniqueName: \"kubernetes.io/projected/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-kube-api-access-xvm9d\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:17.121092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.121034    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:17.121092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.121047    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d0db60-46fc-4929-bcae-f0bb9f2e587b-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:17.399270 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.399179    5787 generic.go:356] "Generic (PLEG): container finished" podID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerID="3dbb6584d4489f31a4bff887a1e76c9829406f5c61aeb47d0a088bea33a39100" exitCode=60
Oct 31 13:15:17.399775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.399265    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" event={"ID":"3736d19a-1161-42bc-bf98-ec9d6408a563","Type":"ContainerDied","Data":"3dbb6584d4489f31a4bff887a1e76c9829406f5c61aeb47d0a088bea33a39100"}
Oct 31 13:15:17.399775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.399336    5787 scope.go:122] "RemoveContainer" containerID="da180e221cf9ec7d0b215b92908bdcaed2757f07ca158b7ec294dfbd9136590a"
Oct 31 13:15:17.399917 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.399818    5787 scope.go:122] "RemoveContainer" containerID="3dbb6584d4489f31a4bff887a1e76c9829406f5c61aeb47d0a088bea33a39100"
Oct 31 13:15:17.400251 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:17.400222    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd_odh-ai-gateway-infra(3736d19a-1161-42bc-bf98-ec9d6408a563)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563"
Oct 31 13:15:17.403147 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.403123    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"
Oct 31 13:15:17.403242 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.403176    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9" event={"ID":"e1d0db60-46fc-4929-bcae-f0bb9f2e587b","Type":"ContainerDied","Data":"76a9ab566c8d07383e87631566db16b5a6128c029594555822e8bbe9fecfbcd5"}
Oct 31 13:15:17.403242 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.403224    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76a9ab566c8d07383e87631566db16b5a6128c029594555822e8bbe9fecfbcd5"
Oct 31 13:15:17.427895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.427861    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"]
Oct 31 13:15:17.564792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.564766    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:17.738087 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.733806    5787 scope.go:122] "RemoveContainer" containerID="3f5ab7cf0d4412e679ee0939e495f7ce2c3c4feb938156e52c299a23c1099e8d"
Oct 31 13:15:17.738087 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.734834    5787 scope.go:122] "RemoveContainer" containerID="10443eb208bf881d45a80eb43058e9449891683578d6bb6934cd7d4ed3fe92db"
Oct 31 13:15:17.774096 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.772974    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b8769e-6fce-4fbd-8765-d7144d767d1c-kube-api-access-lbq8h\" (UniqueName: \"kubernetes.io/projected/f4b8769e-6fce-4fbd-8765-d7144d767d1c-kube-api-access-lbq8h\") pod \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\" (UID: \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\") "
Oct 31 13:15:17.774096 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.773092    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b8769e-6fce-4fbd-8765-d7144d767d1c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b8769e-6fce-4fbd-8765-d7144d767d1c-maas-api-service-ca\") pod \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\" (UID: \"f4b8769e-6fce-4fbd-8765-d7144d767d1c\") "
Oct 31 13:15:17.775658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.775626    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b8769e-6fce-4fbd-8765-d7144d767d1c-maas-api-service-ca" pod "f4b8769e-6fce-4fbd-8765-d7144d767d1c" (UID: "f4b8769e-6fce-4fbd-8765-d7144d767d1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:17.809081 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.796433    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b8769e-6fce-4fbd-8765-d7144d767d1c-kube-api-access-lbq8h" pod "f4b8769e-6fce-4fbd-8765-d7144d767d1c" (UID: "f4b8769e-6fce-4fbd-8765-d7144d767d1c"). InnerVolumeSpecName "kube-api-access-lbq8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:17.875664 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.875639    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbq8h\" (UniqueName: \"kubernetes.io/projected/f4b8769e-6fce-4fbd-8765-d7144d767d1c-kube-api-access-lbq8h\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:17.875664 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.875664    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b8769e-6fce-4fbd-8765-d7144d767d1c-maas-api-service-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:17.952598 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.952555    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"]
Oct 31 13:15:17.953289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953268    5787 cpu_manager.go:414] "containerMap: removing container" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerName="cleanup"
Oct 31 13:15:17.953410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953290    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerName="cleanup"
Oct 31 13:15:17.953410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953315    5787 cpu_manager.go:414] "containerMap: removing container" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerName="cleanup"
Oct 31 13:15:17.953410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953323    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerName="cleanup"
Oct 31 13:15:17.953410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953342    5787 cpu_manager.go:414] "containerMap: removing container" podUID="e1d0db60-46fc-4929-bcae-f0bb9f2e587b" containerName="revoke-keys"
Oct 31 13:15:17.953410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953349    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d0db60-46fc-4929-bcae-f0bb9f2e587b" containerName="revoke-keys"
Oct 31 13:15:17.953410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953380    5787 cpu_manager.go:414] "containerMap: removing container" podUID="8eb9a0f4-b481-496b-b905-e9667799c175" containerName="revoke-keys"
Oct 31 13:15:17.953410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953387    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb9a0f4-b481-496b-b905-e9667799c175" containerName="revoke-keys"
Oct 31 13:15:17.953749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953575    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb9a0f4-b481-496b-b905-e9667799c175" containerName="revoke-keys"
Oct 31 13:15:17.953749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953594    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerName="cleanup"
Oct 31 13:15:17.953749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953607    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerName="cleanup"
Oct 31 13:15:17.953749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953617    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerName="cleanup"
Oct 31 13:15:17.953749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.953627    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d0db60-46fc-4929-bcae-f0bb9f2e587b" containerName="revoke-keys"
Oct 31 13:15:17.958721 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.958685    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:17.964387 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.964362    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:15:17.975896 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.968597    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"]
Oct 31 13:15:17.976977 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.976950    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b62cb9-nczgb\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:17.977089 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.976997    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b62cb9-nczgb\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:17.977318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:17.977298    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnck9\" (UniqueName: \"kubernetes.io/projected/0729f58a-7f4d-4966-90ac-796c49ade823-kube-api-access-qnck9\") pod \"maas-api-revoke-sub-13b62cb9-nczgb\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:18.079247 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.079154    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b62cb9-nczgb\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:18.079247 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.079206    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b62cb9-nczgb\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:18.079535 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.079517    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnck9\" (UniqueName: \"kubernetes.io/projected/0729f58a-7f4d-4966-90ac-796c49ade823-kube-api-access-qnck9\") pod \"maas-api-revoke-sub-13b62cb9-nczgb\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:18.079923 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.079906    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b62cb9-nczgb\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:18.081765 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.081742    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b62cb9-nczgb\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:18.092590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.092564    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnck9\" (UniqueName: \"kubernetes.io/projected/0729f58a-7f4d-4966-90ac-796c49ade823-kube-api-access-qnck9\") pod \"maas-api-revoke-sub-13b62cb9-nczgb\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:18.251484 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:18.251450    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:15:18.278431 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.278403    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:18.416619 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.416592    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"
Oct 31 13:15:18.417086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.416632    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8" event={"ID":"f4b8769e-6fce-4fbd-8765-d7144d767d1c","Type":"ContainerDied","Data":"3b55af145e59cd652196ff28c1d6479803889899c59f7be4ae5550ea78fe62f3"}
Oct 31 13:15:18.417086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.416741    5787 scope.go:122] "RemoveContainer" containerID="48d602964cc39b0cebc63fb610ca4d7825c25929bd72c1ab5bddb8fe852e9022"
Oct 31 13:15:18.420857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.420809    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" event={"ID":"52986769-9ebe-449b-8e8b-1b1e82579282","Type":"ContainerStarted","Data":"2b878a053bd75ed072dc247b756f9864e071ccfcf05b881f1ddcc1d7fdad4530"}
Oct 31 13:15:18.424206 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.424177    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" event={"ID":"9e1fa248-4cdb-4a4d-99e4-d9a259547d84","Type":"ContainerStarted","Data":"34851ce653aeb3325482b2a34b994351b563495e7ae1363e11e0c412235426e0"}
Oct 31 13:15:18.474975 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.474937    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"]
Oct 31 13:15:18.481464 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.481437    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-79c6ff-29890875-hgrt8"]
Oct 31 13:15:18.676438 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.676280    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"]
Oct 31 13:15:18.704196 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.704172    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:18.710051 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.710024    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" path="/var/lib/kubelet/pods/f4b8769e-6fce-4fbd-8765-d7144d767d1c/volumes"
Oct 31 13:15:18.828922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.828891    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3736d19a-1161-42bc-bf98-ec9d6408a563-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736d19a-1161-42bc-bf98-ec9d6408a563-maas-api-service-ca\") pod \"3736d19a-1161-42bc-bf98-ec9d6408a563\" (UID: \"3736d19a-1161-42bc-bf98-ec9d6408a563\") "
Oct 31 13:15:18.829139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.829126    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3736d19a-1161-42bc-bf98-ec9d6408a563-kube-api-access-5mnqd\" (UniqueName: \"kubernetes.io/projected/3736d19a-1161-42bc-bf98-ec9d6408a563-kube-api-access-5mnqd\") pod \"3736d19a-1161-42bc-bf98-ec9d6408a563\" (UID: \"3736d19a-1161-42bc-bf98-ec9d6408a563\") "
Oct 31 13:15:18.829336 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.829313    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3736d19a-1161-42bc-bf98-ec9d6408a563-maas-api-service-ca" pod "3736d19a-1161-42bc-bf98-ec9d6408a563" (UID: "3736d19a-1161-42bc-bf98-ec9d6408a563"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:18.829890 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.829865    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3736d19a-1161-42bc-bf98-ec9d6408a563-maas-api-service-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:18.835676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.835654    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3736d19a-1161-42bc-bf98-ec9d6408a563-kube-api-access-5mnqd" pod "3736d19a-1161-42bc-bf98-ec9d6408a563" (UID: "3736d19a-1161-42bc-bf98-ec9d6408a563"). InnerVolumeSpecName "kube-api-access-5mnqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:18.931464 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:18.931372    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mnqd\" (UniqueName: \"kubernetes.io/projected/3736d19a-1161-42bc-bf98-ec9d6408a563-kube-api-access-5mnqd\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:19.434251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.434220    5787 generic.go:356] "Generic (PLEG): container finished" podID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerID="34851ce653aeb3325482b2a34b994351b563495e7ae1363e11e0c412235426e0" exitCode=60
Oct 31 13:15:19.434756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.434297    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" event={"ID":"9e1fa248-4cdb-4a4d-99e4-d9a259547d84","Type":"ContainerDied","Data":"34851ce653aeb3325482b2a34b994351b563495e7ae1363e11e0c412235426e0"}
Oct 31 13:15:19.434756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.434365    5787 scope.go:122] "RemoveContainer" containerID="10443eb208bf881d45a80eb43058e9449891683578d6bb6934cd7d4ed3fe92db"
Oct 31 13:15:19.434899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.434870    5787 scope.go:122] "RemoveContainer" containerID="34851ce653aeb3325482b2a34b994351b563495e7ae1363e11e0c412235426e0"
Oct 31 13:15:19.435355 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:19.435317    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw_odh-ai-gateway-infra(9e1fa248-4cdb-4a4d-99e4-d9a259547d84)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84"
Oct 31 13:15:19.436337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.436314    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb" event={"ID":"0729f58a-7f4d-4966-90ac-796c49ade823","Type":"ContainerStarted","Data":"d862c33421da9249aa5cfff45c509d611f59e9a812d41d34473f2a10d5161fa9"}
Oct 31 13:15:19.436459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.436369    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb" event={"ID":"0729f58a-7f4d-4966-90ac-796c49ade823","Type":"ContainerStarted","Data":"90bfbb1126cc3b67062b05b63bbf66d58405aa4b19a32f49e3d4b288396665fb"}
Oct 31 13:15:19.437894 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.437837    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"
Oct 31 13:15:19.437894 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.437858    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd" event={"ID":"3736d19a-1161-42bc-bf98-ec9d6408a563","Type":"ContainerDied","Data":"ad4cbaf13704f790376ce167ba5698240651f80f801d5869181077dabd35d9d1"}
Oct 31 13:15:19.441309 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.441285    5787 generic.go:356] "Generic (PLEG): container finished" podID="52986769-9ebe-449b-8e8b-1b1e82579282" containerID="2b878a053bd75ed072dc247b756f9864e071ccfcf05b881f1ddcc1d7fdad4530" exitCode=60
Oct 31 13:15:19.441399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.441354    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" event={"ID":"52986769-9ebe-449b-8e8b-1b1e82579282","Type":"ContainerDied","Data":"2b878a053bd75ed072dc247b756f9864e071ccfcf05b881f1ddcc1d7fdad4530"}
Oct 31 13:15:19.441681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.441668    5787 scope.go:122] "RemoveContainer" containerID="2b878a053bd75ed072dc247b756f9864e071ccfcf05b881f1ddcc1d7fdad4530"
Oct 31 13:15:19.441985 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:19.441968    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l_odh-ai-gateway-infra(52986769-9ebe-449b-8e8b-1b1e82579282)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" podUID="52986769-9ebe-449b-8e8b-1b1e82579282"
Oct 31 13:15:19.448818 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.448797    5787 scope.go:122] "RemoveContainer" containerID="3dbb6584d4489f31a4bff887a1e76c9829406f5c61aeb47d0a088bea33a39100"
Oct 31 13:15:19.458495 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.458477    5787 scope.go:122] "RemoveContainer" containerID="3f5ab7cf0d4412e679ee0939e495f7ce2c3c4feb938156e52c299a23c1099e8d"
Oct 31 13:15:19.510385 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.510334    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb" podStartSLOduration=2.510318979 podStartE2EDuration="2.510318979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:19.506814381 +0000 UTC m=+898.922756222" watchObservedRunningTime="2026-10-31 13:15:19.510318979 +0000 UTC m=+898.926260865"
Oct 31 13:15:19.519267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.519238    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"]
Oct 31 13:15:19.527936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.527907    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"]
Oct 31 13:15:19.534681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.534650    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-426787-29890875-xfncd"]
Oct 31 13:15:19.545233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:19.545196    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"]
Oct 31 13:15:20.376787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.376755    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"]
Oct 31 13:15:20.377478 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377459    5787 cpu_manager.go:414] "containerMap: removing container" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerName="cleanup"
Oct 31 13:15:20.377605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377479    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b8769e-6fce-4fbd-8765-d7144d767d1c" containerName="cleanup"
Oct 31 13:15:20.377605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377536    5787 cpu_manager.go:414] "containerMap: removing container" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerName="cleanup"
Oct 31 13:15:20.377605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377543    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerName="cleanup"
Oct 31 13:15:20.377605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377561    5787 cpu_manager.go:414] "containerMap: removing container" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerName="cleanup"
Oct 31 13:15:20.377605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377568    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerName="cleanup"
Oct 31 13:15:20.377605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377600    5787 cpu_manager.go:414] "containerMap: removing container" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerName="cleanup"
Oct 31 13:15:20.377913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377607    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerName="cleanup"
Oct 31 13:15:20.377913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377779    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerName="cleanup"
Oct 31 13:15:20.377913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.377794    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerName="cleanup"
Oct 31 13:15:20.381622 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.381604    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.391888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.391861    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"]
Oct 31 13:15:20.459479 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.459451    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.459953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.459661    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54nw\" (UniqueName: \"kubernetes.io/projected/5817ea86-18de-470f-b84e-8ff152d98f61-kube-api-access-c54nw\") pod \"maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.459953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.459735    5787 generic.go:356] "Generic (PLEG): container finished" podID="0729f58a-7f4d-4966-90ac-796c49ade823" containerID="d862c33421da9249aa5cfff45c509d611f59e9a812d41d34473f2a10d5161fa9" exitCode=0
Oct 31 13:15:20.459953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.459794    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.459953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.459839    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb" event={"ID":"0729f58a-7f4d-4966-90ac-796c49ade823","Type":"ContainerDied","Data":"d862c33421da9249aa5cfff45c509d611f59e9a812d41d34473f2a10d5161fa9"}
Oct 31 13:15:20.509114 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.509076    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"]
Oct 31 13:15:20.511683 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.511654    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" containerName="cleanup"
Oct 31 13:15:20.525257 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.525233    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:20.556865 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.553581    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"]
Oct 31 13:15:20.565682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.565639    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54nw\" (UniqueName: \"kubernetes.io/projected/5817ea86-18de-470f-b84e-8ff152d98f61-kube-api-access-c54nw\") pod \"maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.565872 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.565843    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.571786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.566676    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.571786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.567581    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.571786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.571647    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.587468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.587430    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54nw\" (UniqueName: \"kubernetes.io/projected/5817ea86-18de-470f-b84e-8ff152d98f61-kube-api-access-c54nw\") pod \"maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.589564 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.589536    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"]
Oct 31 13:15:20.594492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.594470    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.619272 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.619240    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"]
Oct 31 13:15:20.702406 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.702367    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9bfd38d-6jhv2\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.702783 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.702755    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65kqb\" (UniqueName: \"kubernetes.io/projected/efde1a18-f86c-4247-bd41-9ac23d244500-kube-api-access-65kqb\") pod \"maas-api-revoke-sub-c330c431-zg9lr\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:20.706947 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.706696    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dbqb\" (UniqueName: \"kubernetes.io/projected/6460e2c0-e9ba-4943-aee0-c06f74ede22d-kube-api-access-6dbqb\") pod \"maas-api-revoke-sub-c9bfd38d-6jhv2\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.707211 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.707190    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9bfd38d-6jhv2\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.707884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.707859    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c330c431-zg9lr\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:20.708103 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.708087    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c330c431-zg9lr\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:20.712390 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.712372    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:20.840730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.837757    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9bfd38d-6jhv2\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.840730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.837912    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65kqb\" (UniqueName: \"kubernetes.io/projected/efde1a18-f86c-4247-bd41-9ac23d244500-kube-api-access-65kqb\") pod \"maas-api-revoke-sub-c330c431-zg9lr\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:20.840730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.837943    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dbqb\" (UniqueName: \"kubernetes.io/projected/6460e2c0-e9ba-4943-aee0-c06f74ede22d-kube-api-access-6dbqb\") pod \"maas-api-revoke-sub-c9bfd38d-6jhv2\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.840730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.837983    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9bfd38d-6jhv2\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.840730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.838074    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c330c431-zg9lr\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:20.840730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.838163    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c330c431-zg9lr\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:20.840730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.839310    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c330c431-zg9lr\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:20.840730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.840176    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9bfd38d-6jhv2\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.929335 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.915231    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9bfd38d-6jhv2\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.948731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.934457    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dbqb\" (UniqueName: \"kubernetes.io/projected/6460e2c0-e9ba-4943-aee0-c06f74ede22d-kube-api-access-6dbqb\") pod \"maas-api-revoke-sub-c9bfd38d-6jhv2\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:20.975907 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.975170    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3736d19a-1161-42bc-bf98-ec9d6408a563" path="/var/lib/kubelet/pods/3736d19a-1161-42bc-bf98-ec9d6408a563/volumes"
Oct 31 13:15:20.975907 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:20.975811    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:21.006917 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.002512    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65kqb\" (UniqueName: \"kubernetes.io/projected/efde1a18-f86c-4247-bd41-9ac23d244500-kube-api-access-65kqb\") pod \"maas-api-revoke-sub-c330c431-zg9lr\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:21.047779 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.020450    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c330c431-zg9lr\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:21.072413 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.072379    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52986769-9ebe-449b-8e8b-1b1e82579282-kube-api-access-l6t8g\" (UniqueName: \"kubernetes.io/projected/52986769-9ebe-449b-8e8b-1b1e82579282-kube-api-access-l6t8g\") pod \"52986769-9ebe-449b-8e8b-1b1e82579282\" (UID: \"52986769-9ebe-449b-8e8b-1b1e82579282\") "
Oct 31 13:15:21.109048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.108795    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:21.143731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.121893    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52986769-9ebe-449b-8e8b-1b1e82579282-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52986769-9ebe-449b-8e8b-1b1e82579282-maas-api-service-ca\") pod \"52986769-9ebe-449b-8e8b-1b1e82579282\" (UID: \"52986769-9ebe-449b-8e8b-1b1e82579282\") "
Oct 31 13:15:21.143731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.123082    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52986769-9ebe-449b-8e8b-1b1e82579282-maas-api-service-ca" pod "52986769-9ebe-449b-8e8b-1b1e82579282" (UID: "52986769-9ebe-449b-8e8b-1b1e82579282"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:21.159555 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.158643    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:21.183743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.183370    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:21.245013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.244977    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52986769-9ebe-449b-8e8b-1b1e82579282-maas-api-service-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:21.254557 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.249904    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52986769-9ebe-449b-8e8b-1b1e82579282-kube-api-access-l6t8g" pod "52986769-9ebe-449b-8e8b-1b1e82579282" (UID: "52986769-9ebe-449b-8e8b-1b1e82579282"). InnerVolumeSpecName "kube-api-access-l6t8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:21.254743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.254649    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" containerID="cri-o://33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962" gracePeriod=45
Oct 31 13:15:21.296974 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.296931    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" containerID="cri-o://2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0" gracePeriod=45
Oct 31 13:15:21.308263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.308226    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" containerID="cri-o://699973dca7537958c1cf523e6274bde1aa94bccac02682d9644f47942ceba0a7" gracePeriod=45
Oct 31 13:15:21.361729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.352145    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-kube-api-access-wnz88\" (UniqueName: \"kubernetes.io/projected/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-kube-api-access-wnz88\") pod \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\" (UID: \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\") "
Oct 31 13:15:21.361729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.352288    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-maas-api-service-ca\") pod \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\" (UID: \"9e1fa248-4cdb-4a4d-99e4-d9a259547d84\") "
Oct 31 13:15:21.361729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.353038    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6t8g\" (UniqueName: \"kubernetes.io/projected/52986769-9ebe-449b-8e8b-1b1e82579282-kube-api-access-l6t8g\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:21.444742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.376546    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-maas-api-service-ca" pod "9e1fa248-4cdb-4a4d-99e4-d9a259547d84" (UID: "9e1fa248-4cdb-4a4d-99e4-d9a259547d84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:21.444742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.401960    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-kube-api-access-wnz88" pod "9e1fa248-4cdb-4a4d-99e4-d9a259547d84" (UID: "9e1fa248-4cdb-4a4d-99e4-d9a259547d84"). InnerVolumeSpecName "kube-api-access-wnz88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:21.500016 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.456043    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-maas-api-service-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:21.500016 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.456081    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnz88\" (UniqueName: \"kubernetes.io/projected/9e1fa248-4cdb-4a4d-99e4-d9a259547d84-kube-api-access-wnz88\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:21.666856 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.614825    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused"
Oct 31 13:15:21.770763 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:21.752098    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 31 13:15:22.035192 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.014628    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l" event={"ID":"52986769-9ebe-449b-8e8b-1b1e82579282","Type":"ContainerDied","Data":"46d22a0ad43be3ab13d82057b3ba84c63a6e14fd8bfbb9fdb233d9ef34c8d4f4"}
Oct 31 13:15:22.035192 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.014730    5787 scope.go:122] "RemoveContainer" containerID="2b878a053bd75ed072dc247b756f9864e071ccfcf05b881f1ddcc1d7fdad4530"
Oct 31 13:15:22.035192 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.015195    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"
Oct 31 13:15:22.114730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.051616    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"]
Oct 31 13:15:22.141519 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.141084    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"
Oct 31 13:15:22.141842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.141573    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw" event={"ID":"9e1fa248-4cdb-4a4d-99e4-d9a259547d84","Type":"ContainerDied","Data":"9cb8a80d58b0646091f3e8a3a0188ef51a44d0edc39244b433519226e97835f6"}
Oct 31 13:15:22.455100 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.451040    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": dial tcp 10.134.0.38:8000: connect: connection refused"
Oct 31 13:15:22.697732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.677503    5787 scope.go:122] "RemoveContainer" containerID="34851ce653aeb3325482b2a34b994351b563495e7ae1363e11e0c412235426e0"
Oct 31 13:15:22.698443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.691733    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:22.752777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.750568    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:22.785857 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:22.767628    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5817ea86_18de_470f_b84e_8ff152d98f61.slice/crio-cf8e04951f418280f87ecf3f2074327aa4d153016da96d0704229abe6052d5ff WatchSource:0}: Error finding container cf8e04951f418280f87ecf3f2074327aa4d153016da96d0704229abe6052d5ff: Status 404 returned error can't find the container with id cf8e04951f418280f87ecf3f2074327aa4d153016da96d0704229abe6052d5ff
Oct 31 13:15:22.817147 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.808767    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 31 13:15:22.817147 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.808952    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:15:22.817147 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.809376    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused"
Oct 31 13:15:22.825184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.819148    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:15:22.942733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.895827    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:15:22.942733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:22.896318    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:23.391733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.373275    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:23.447274 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.445268    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:23.448568 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.448532    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:23.655167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.655122    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:15:23.699953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.699786    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:23.751910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.751869    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused"
Oct 31 13:15:23.752262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.752230    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": dial tcp 10.134.0.38:8000: connect: connection refused"
Oct 31 13:15:23.799428 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.794267    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/health\": context deadline exceeded"
Oct 31 13:15:23.799428 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.794359    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded"
Oct 31 13:15:23.842261 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.840884    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 31 13:15:23.901101 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.899357    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:15:23.985948 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:23.985904    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:24.218335 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.218293    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 31 13:15:24.246801 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:24.246769    5787 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.427s"
Oct 31 13:15:24.341686 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.341649    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": dial tcp 10.134.0.38:8000: connect: connection refused"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.376199    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.400639    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h" event={"ID":"5817ea86-18de-470f-b84e-8ff152d98f61","Type":"ContainerStarted","Data":"cf8e04951f418280f87ecf3f2074327aa4d153016da96d0704229abe6052d5ff"}
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.400743    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.400786    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.400806    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"]
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.401796    5787 cpu_manager.go:414] "containerMap: removing container" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.401817    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.401867    5787 cpu_manager.go:414] "containerMap: removing container" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.401875    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.401886    5787 cpu_manager.go:414] "containerMap: removing container" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.401893    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.401906    5787 cpu_manager.go:414] "containerMap: removing container" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.401912    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.402129    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.402144    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.402152    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.402165    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" containerName="cleanup"
Oct 31 13:15:24.416130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.402182    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerName="cleanup"
Oct 31 13:15:24.501798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.436876    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:24.501798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.463722    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:24.564081 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.529839    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:24.567431 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.565721    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused"
Oct 31 13:15:24.752765 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.702539    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 31 13:15:24.752765 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.737628    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 31 13:15:24.812403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.812357    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:24.896034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.888925    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:24.909993 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:24.909952    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:15:25.025494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.025447    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:25.111614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.025796    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:25.288916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.265651    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 31 13:15:25.394808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.393572    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:25.438394 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.438350    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:15:25.461628 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:25.455607    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefde1a18_f86c_4247_bd41_9ac23d244500.slice/crio-ce590d597a818fe632983256bce6523ef29060b182c24abee6f963cba09f8d99 WatchSource:0}: Error finding container ce590d597a818fe632983256bce6523ef29060b182c24abee6f963cba09f8d99: Status 404 returned error can't find the container with id ce590d597a818fe632983256bce6523ef29060b182c24abee6f963cba09f8d99
Oct 31 13:15:25.461628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.456330    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": dial tcp 10.134.0.38:8000: connect: connection refused"
Oct 31 13:15:25.487520 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.487481    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:25.580044 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.579939    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:25.580899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.580864    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:15:25.610795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.601618    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused"
Oct 31 13:15:25.730498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.725268    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:25.779179 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.744725    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 31 13:15:25.855998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.855866    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:25.902559 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:25.902511    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:26.085858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.085807    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:26.146838 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.119795    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:26.276213 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.276166    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" containerID="cri-o://13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb" gracePeriod=45
Oct 31 13:15:26.276731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.276525    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" containerID="cri-o://8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7" gracePeriod=45
Oct 31 13:15:26.298356 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.298314    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:26.339925 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.339879    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": dial tcp 10.134.0.38:8000: connect: connection refused"
Oct 31 13:15:26.398285 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.398185    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:26.466337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.465074    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": read tcp 10.134.0.2:47378->10.134.0.70:8000: read: connection reset by peer"
Oct 31 13:15:26.478762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.471099    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:26.508378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.507173    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:26.578110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.573068    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused"
Oct 31 13:15:26.677606 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.662498    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": context deadline exceeded"
Oct 31 13:15:26.745111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.745060    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 31 13:15:26.746181 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.746139    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:26.746744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.746697    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 31 13:15:26.857853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:26.857805    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:27.381219 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.379675    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 31 13:15:27.383476 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.383441    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Oct 31 13:15:27.387793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.387579    5787 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:27.409893 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.409854    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": dial tcp 10.134.0.38:8000: connect: connection refused"
Oct 31 13:15:27.410972 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.410947    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused"
Oct 31 13:15:27.426617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.411226    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": dial tcp 10.134.0.77:8000: connect: connection refused"
Oct 31 13:15:27.433005 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.426788    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": context deadline exceeded"
Oct 31 13:15:27.439054 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.426860    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 31 13:15:27.458757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.440444    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"]
Oct 31 13:15:27.458757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.440535    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:15:27.480791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.479140    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:27.485611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.481494    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:27.485611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.483522    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:15:27.558901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.538798    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:15:28.002305 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:27.997687    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:28.008356 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.008160    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:28.029960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.025152    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0729f58a-7f4d-4966-90ac-796c49ade823-kube-api-access-qnck9\" (UniqueName: \"kubernetes.io/projected/0729f58a-7f4d-4966-90ac-796c49ade823-kube-api-access-qnck9\") pod \"0729f58a-7f4d-4966-90ac-796c49ade823\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") "
Oct 31 13:15:28.042400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.037084    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.082668    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb" event={"ID":"0729f58a-7f4d-4966-90ac-796c49ade823","Type":"ContainerDied","Data":"90bfbb1126cc3b67062b05b63bbf66d58405aa4b19a32f49e3d4b288396665fb"}
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.082741    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90bfbb1126cc3b67062b05b63bbf66d58405aa4b19a32f49e3d4b288396665fb"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.082805    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.082826    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2" event={"ID":"6460e2c0-e9ba-4943-aee0-c06f74ede22d","Type":"ContainerStarted","Data":"8c969a17b894342d949c9191fc5cf2396561042224a10c136d665221fb5fa906"}
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.082876    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.082909    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.082946    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.082963    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"]
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083022    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083040    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a2c006-29890875-hsz2l"]
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083082    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083104    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"]
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083124    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"]
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083179    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083198    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7bf66c-29890875-ppdkw"]
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083236    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083262    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"]
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083284    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"]
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083329    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:15:28.084609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.083367    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:15:28.105982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.096172    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": dial tcp 10.134.0.77:8000: connect: connection refused"
Oct 31 13:15:28.105982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.096821    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-xks-ca\") pod \"0729f58a-7f4d-4966-90ac-796c49ade823\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") "
Oct 31 13:15:28.105982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.096890    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-openshift-ca\") pod \"0729f58a-7f4d-4966-90ac-796c49ade823\" (UID: \"0729f58a-7f4d-4966-90ac-796c49ade823\") "
Oct 31 13:15:28.105982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.097454    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.114026    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.114257    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5m94\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-kube-api-access-v5m94\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.114408    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.114542    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.114573    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fe87e5-d534-4f37-8260-5d5b415df0ff-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.114596    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.114630    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.114930    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.127861    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-xks-ca" pod "0729f58a-7f4d-4966-90ac-796c49ade823" (UID: "0729f58a-7f4d-4966-90ac-796c49ade823"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.128173    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-openshift-ca" pod "0729f58a-7f4d-4966-90ac-796c49ade823" (UID: "0729f58a-7f4d-4966-90ac-796c49ade823"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.134568    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Oct 31 13:15:28.137968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.134644    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": dial tcp 10.134.0.42:8000: connect: connection refused"
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.152730    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0729f58a-7f4d-4966-90ac-796c49ade823-kube-api-access-qnck9" pod "0729f58a-7f4d-4966-90ac-796c49ade823" (UID: "0729f58a-7f4d-4966-90ac-796c49ade823"). InnerVolumeSpecName "kube-api-access-qnck9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.152789    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"]
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.153758    5787 cpu_manager.go:414] "containerMap: removing container" podUID="0729f58a-7f4d-4966-90ac-796c49ade823" containerName="revoke-keys"
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.153773    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0729f58a-7f4d-4966-90ac-796c49ade823" containerName="revoke-keys"
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.153818    5787 cpu_manager.go:414] "containerMap: removing container" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerName="cleanup"
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.153826    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerName="cleanup"
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.153838    5787 cpu_manager.go:414] "containerMap: removing container" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" containerName="cleanup"
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.153846    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" containerName="cleanup"
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.153991    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" containerName="cleanup"
Oct 31 13:15:28.154777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.154005    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="0729f58a-7f4d-4966-90ac-796c49ade823" containerName="revoke-keys"
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.169968    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.170006    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"]
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.170064    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.170096    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.170125    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.170204    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.171296    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.171334    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" event={"ID":"d366feca-7052-4618-8a80-b929afe65e16","Type":"ContainerDied","Data":"699973dca7537958c1cf523e6274bde1aa94bccac02682d9644f47942ceba0a7"}
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.171290    5787 generic.go:356] "Generic (PLEG): container finished" podID="d366feca-7052-4618-8a80-b929afe65e16" containerID="699973dca7537958c1cf523e6274bde1aa94bccac02682d9644f47942ceba0a7" exitCode=0
Oct 31 13:15:28.196304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.188348    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:15:28.227199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.223684    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5m94\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-kube-api-access-v5m94\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.227199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.223809    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.227199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.223947    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.227199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.223978    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fe87e5-d534-4f37-8260-5d5b415df0ff-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.227199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.224019    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.227199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.224053    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.228006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.227892    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.228343 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.228291    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.229004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.228542    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.229004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.228840    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnck9\" (UniqueName: \"kubernetes.io/projected/0729f58a-7f4d-4966-90ac-796c49ade823-kube-api-access-qnck9\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.229004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.228872    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.229004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.228912    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0729f58a-7f4d-4966-90ac-796c49ade823-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.230675 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.230291    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-credential-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.237681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.233612    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-data\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.237681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.236129    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fe87e5-d534-4f37-8260-5d5b415df0ff-istiod-ca-cert\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.237681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.236662    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-certs\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.237681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.237007    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-socket\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.269260 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:28.265265    5787 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf8cdef9_566f_443d_9949_3a8e9947fd4e.slice/crio-13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf8cdef9_566f_443d_9949_3a8e9947fd4e.slice/crio-conmon-13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc65d5f1f_befb_4c32_8f0e_0a09f4915b53.slice/crio-conmon-8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc43584fa_29e8_4ab5_ab4e_366ed24bade8.slice/crio-conmon-2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:15:28.281106 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.280960    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": dial tcp 10.134.0.70:8000: connect: connection refused"
Oct 31 13:15:28.331351 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.331048    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.331724 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.331587    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.331724 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.331669    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j6h8\" (UniqueName: \"kubernetes.io/projected/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-kube-api-access-6j6h8\") pod \"maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.337953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.337222    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5m94\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-kube-api-access-v5m94\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.339960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.339783    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-envoy\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.339960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.339906    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-podinfo\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.340177 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.340026    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-token\") pod \"e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") " pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.389637 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.389603    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:15:28.397086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.396928    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:15:28.402870 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.402837    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:15:28.433121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.433089    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.433318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.433139    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j6h8\" (UniqueName: \"kubernetes.io/projected/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-kube-api-access-6j6h8\") pod \"maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.433318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.433222    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.435301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.434885    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.455984 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.455946    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j6h8\" (UniqueName: \"kubernetes.io/projected/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-kube-api-access-6j6h8\") pod \"maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.477085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.477009    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.502155 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.502120    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:28.533846 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.533820    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d366feca-7052-4618-8a80-b929afe65e16-tls-certs\" (UniqueName: \"kubernetes.io/secret/d366feca-7052-4618-8a80-b929afe65e16-tls-certs\") pod \"d366feca-7052-4618-8a80-b929afe65e16\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") "
Oct 31 13:15:28.534007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.533875    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-model-cache\") pod \"d366feca-7052-4618-8a80-b929afe65e16\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") "
Oct 31 13:15:28.534007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.533918    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-dshm\") pod \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") "
Oct 31 13:15:28.534007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.533962    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-model-cache\") pod \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") "
Oct 31 13:15:28.534007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534005    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-home\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-home\") pod \"d366feca-7052-4618-8a80-b929afe65e16\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") "
Oct 31 13:15:28.534319 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534054    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-home\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-home\") pod \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") "
Oct 31 13:15:28.534319 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534104    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d366feca-7052-4618-8a80-b929afe65e16-kube-api-access-ln4nl\" (UniqueName: \"kubernetes.io/projected/d366feca-7052-4618-8a80-b929afe65e16-kube-api-access-ln4nl\") pod \"d366feca-7052-4618-8a80-b929afe65e16\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") "
Oct 31 13:15:28.534319 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534137    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-dshm\") pod \"d366feca-7052-4618-8a80-b929afe65e16\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") "
Oct 31 13:15:28.534319 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534223    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-dshm\") pod \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") "
Oct 31 13:15:28.534319 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534289    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-home\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-home\") pod \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") "
Oct 31 13:15:28.534665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534330    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-tmp-dir\") pod \"d366feca-7052-4618-8a80-b929afe65e16\" (UID: \"d366feca-7052-4618-8a80-b929afe65e16\") "
Oct 31 13:15:28.534665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534377    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tmp-dir\") pod \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") "
Oct 31 13:15:28.534665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534417    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-model-cache\") pod \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") "
Oct 31 13:15:28.534665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534490    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tls-certs\" (UniqueName: \"kubernetes.io/secret/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tls-certs\") pod \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") "
Oct 31 13:15:28.534665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534521    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c43584fa-29e8-4ab5-ab4e-366ed24bade8-kube-api-access-c8frn\" (UniqueName: \"kubernetes.io/projected/c43584fa-29e8-4ab5-ab4e-366ed24bade8-kube-api-access-c8frn\") pod \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") "
Oct 31 13:15:28.534665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534579    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-kube-api-access-sh52l\" (UniqueName: \"kubernetes.io/projected/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-kube-api-access-sh52l\") pod \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") "
Oct 31 13:15:28.535453 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534673    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tls-certs\" (UniqueName: \"kubernetes.io/secret/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tls-certs\") pod \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\" (UID: \"c43584fa-29e8-4ab5-ab4e-366ed24bade8\") "
Oct 31 13:15:28.535453 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.534729    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tmp-dir\") pod \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\" (UID: \"04a8e1d5-8801-435a-a98e-e4ae80cf10a6\") "
Oct 31 13:15:28.536278 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.536233    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tmp-dir" pod "04a8e1d5-8801-435a-a98e-e4ae80cf10a6" (UID: "04a8e1d5-8801-435a-a98e-e4ae80cf10a6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.539001 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.538790    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-model-cache" pod "c43584fa-29e8-4ab5-ab4e-366ed24bade8" (UID: "c43584fa-29e8-4ab5-ab4e-366ed24bade8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.539301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.539140    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-home" pod "c43584fa-29e8-4ab5-ab4e-366ed24bade8" (UID: "c43584fa-29e8-4ab5-ab4e-366ed24bade8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.539603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.539451    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-tmp-dir" pod "d366feca-7052-4618-8a80-b929afe65e16" (UID: "d366feca-7052-4618-8a80-b929afe65e16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.540863 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.540794    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tmp-dir" pod "c43584fa-29e8-4ab5-ab4e-366ed24bade8" (UID: "c43584fa-29e8-4ab5-ab4e-366ed24bade8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.541984 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.541020    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-home" pod "d366feca-7052-4618-8a80-b929afe65e16" (UID: "d366feca-7052-4618-8a80-b929afe65e16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.541984 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.541196    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-model-cache" pod "d366feca-7052-4618-8a80-b929afe65e16" (UID: "d366feca-7052-4618-8a80-b929afe65e16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.546017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.543197    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d366feca-7052-4618-8a80-b929afe65e16-tls-certs" pod "d366feca-7052-4618-8a80-b929afe65e16" (UID: "d366feca-7052-4618-8a80-b929afe65e16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:28.546017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.543461    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-home" pod "04a8e1d5-8801-435a-a98e-e4ae80cf10a6" (UID: "04a8e1d5-8801-435a-a98e-e4ae80cf10a6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.546017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.545668    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-model-cache" pod "04a8e1d5-8801-435a-a98e-e4ae80cf10a6" (UID: "04a8e1d5-8801-435a-a98e-e4ae80cf10a6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.570418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.565176    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-dshm" pod "d366feca-7052-4618-8a80-b929afe65e16" (UID: "d366feca-7052-4618-8a80-b929afe65e16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.570418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.565350    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-dshm" pod "c43584fa-29e8-4ab5-ab4e-366ed24bade8" (UID: "c43584fa-29e8-4ab5-ab4e-366ed24bade8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.570418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.565439    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tls-certs" pod "c43584fa-29e8-4ab5-ab4e-366ed24bade8" (UID: "c43584fa-29e8-4ab5-ab4e-366ed24bade8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:28.570418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.565488    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d366feca-7052-4618-8a80-b929afe65e16-kube-api-access-ln4nl" pod "d366feca-7052-4618-8a80-b929afe65e16" (UID: "d366feca-7052-4618-8a80-b929afe65e16"). InnerVolumeSpecName "kube-api-access-ln4nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:28.570418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.565591    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tls-certs" pod "04a8e1d5-8801-435a-a98e-e4ae80cf10a6" (UID: "04a8e1d5-8801-435a-a98e-e4ae80cf10a6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:28.570418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.565604    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-kube-api-access-sh52l" pod "04a8e1d5-8801-435a-a98e-e4ae80cf10a6" (UID: "04a8e1d5-8801-435a-a98e-e4ae80cf10a6"). InnerVolumeSpecName "kube-api-access-sh52l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:28.570418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.565642    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-dshm" pod "04a8e1d5-8801-435a-a98e-e4ae80cf10a6" (UID: "04a8e1d5-8801-435a-a98e-e4ae80cf10a6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.570418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.565680    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c43584fa-29e8-4ab5-ab4e-366ed24bade8-kube-api-access-c8frn" pod "c43584fa-29e8-4ab5-ab4e-366ed24bade8" (UID: "c43584fa-29e8-4ab5-ab4e-366ed24bade8"). InnerVolumeSpecName "kube-api-access-c8frn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:28.593302 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.593274    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.643602    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln4nl\" (UniqueName: \"kubernetes.io/projected/d366feca-7052-4618-8a80-b929afe65e16-kube-api-access-ln4nl\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644494    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644532    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644545    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644586    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644601    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644621    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644635    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644650    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8frn\" (UniqueName: \"kubernetes.io/projected/c43584fa-29e8-4ab5-ab4e-366ed24bade8-kube-api-access-c8frn\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644664    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh52l\" (UniqueName: \"kubernetes.io/projected/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-kube-api-access-sh52l\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644683    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c43584fa-29e8-4ab5-ab4e-366ed24bade8-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644717    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644736    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d366feca-7052-4618-8a80-b929afe65e16-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644756    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644782    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c43584fa-29e8-4ab5-ab4e-366ed24bade8-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644797    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644811    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d366feca-7052-4618-8a80-b929afe65e16-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.645586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.644825    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04a8e1d5-8801-435a-a98e-e4ae80cf10a6-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.714007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.713983    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52986769-9ebe-449b-8e8b-1b1e82579282" path="/var/lib/kubelet/pods/52986769-9ebe-449b-8e8b-1b1e82579282/volumes"
Oct 31 13:15:28.715161 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.715144    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1fa248-4cdb-4a4d-99e4-d9a259547d84" path="/var/lib/kubelet/pods/9e1fa248-4cdb-4a4d-99e4-d9a259547d84/volumes"
Oct 31 13:15:28.735599 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.735200    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.867139    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-model-cache\") pod \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") "
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.867311    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tls-certs\") pod \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") "
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.867429    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8cdef9-566f-443d-9949-3a8e9947fd4e-kube-api-access-mpcqz\" (UniqueName: \"kubernetes.io/projected/bf8cdef9-566f-443d-9949-3a8e9947fd4e-kube-api-access-mpcqz\") pod \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") "
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.867506    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-dshm\") pod \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") "
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.867539    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tmp-dir\") pod \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") "
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.867662    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-home\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-home\") pod \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\" (UID: \"bf8cdef9-566f-443d-9949-3a8e9947fd4e\") "
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.871988    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-home" pod "bf8cdef9-566f-443d-9949-3a8e9947fd4e" (UID: "bf8cdef9-566f-443d-9949-3a8e9947fd4e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.872880    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-model-cache" pod "bf8cdef9-566f-443d-9949-3a8e9947fd4e" (UID: "bf8cdef9-566f-443d-9949-3a8e9947fd4e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.873118    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tmp-dir" pod "bf8cdef9-566f-443d-9949-3a8e9947fd4e" (UID: "bf8cdef9-566f-443d-9949-3a8e9947fd4e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.879464    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tls-certs" pod "bf8cdef9-566f-443d-9949-3a8e9947fd4e" (UID: "bf8cdef9-566f-443d-9949-3a8e9947fd4e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.880110    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8cdef9-566f-443d-9949-3a8e9947fd4e-kube-api-access-mpcqz" pod "bf8cdef9-566f-443d-9949-3a8e9947fd4e" (UID: "bf8cdef9-566f-443d-9949-3a8e9947fd4e"). InnerVolumeSpecName "kube-api-access-mpcqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:28.897017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.884413    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-dshm" pod "bf8cdef9-566f-443d-9949-3a8e9947fd4e" (UID: "bf8cdef9-566f-443d-9949-3a8e9947fd4e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:28.903517 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.903480    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"]
Oct 31 13:15:28.972836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.972692    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.972836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.972752    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.972836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.972766    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.972836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.972781    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpcqz\" (UniqueName: \"kubernetes.io/projected/bf8cdef9-566f-443d-9949-3a8e9947fd4e-kube-api-access-mpcqz\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.972836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.972796    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:28.972836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:28.972809    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf8cdef9-566f-443d-9949-3a8e9947fd4e-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:29.008445 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.008017    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:15:29.073856 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.073825    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tls-certs\") pod \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") "
Oct 31 13:15:29.074019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.073886    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-dshm\") pod \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") "
Oct 31 13:15:29.074019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.073910    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-model-cache\") pod \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") "
Oct 31 13:15:29.074019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.073948    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-kube-api-access-67r57\" (UniqueName: \"kubernetes.io/projected/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-kube-api-access-67r57\") pod \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") "
Oct 31 13:15:29.074130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.074058    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-home\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-home\") pod \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") "
Oct 31 13:15:29.074130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.074076    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tmp-dir\") pod \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\" (UID: \"c65d5f1f-befb-4c32-8f0e-0a09f4915b53\") "
Oct 31 13:15:29.074387 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.074353    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-home" pod "c65d5f1f-befb-4c32-8f0e-0a09f4915b53" (UID: "c65d5f1f-befb-4c32-8f0e-0a09f4915b53"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:29.074387 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.074375    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-model-cache" pod "c65d5f1f-befb-4c32-8f0e-0a09f4915b53" (UID: "c65d5f1f-befb-4c32-8f0e-0a09f4915b53"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:29.074570 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.074466    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tmp-dir" pod "c65d5f1f-befb-4c32-8f0e-0a09f4915b53" (UID: "c65d5f1f-befb-4c32-8f0e-0a09f4915b53"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:29.075113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.075084    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:29.075227 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.075118    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:29.075227 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.075135    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:29.134996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.134958    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-dshm" pod "c65d5f1f-befb-4c32-8f0e-0a09f4915b53" (UID: "c65d5f1f-befb-4c32-8f0e-0a09f4915b53"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:29.135301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.135227    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tls-certs" pod "c65d5f1f-befb-4c32-8f0e-0a09f4915b53" (UID: "c65d5f1f-befb-4c32-8f0e-0a09f4915b53"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:29.135301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.135250    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-kube-api-access-67r57" pod "c65d5f1f-befb-4c32-8f0e-0a09f4915b53" (UID: "c65d5f1f-befb-4c32-8f0e-0a09f4915b53"). InnerVolumeSpecName "kube-api-access-67r57". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:29.139916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.139883    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"]
Oct 31 13:15:29.140363 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:29.140325    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27fe87e5_d534_4f37_8260_5d5b415df0ff.slice/crio-4addc61a5efad3af12e50b1d709423daefbe972ffd3d34837c39a248634f5f52 WatchSource:0}: Error finding container 4addc61a5efad3af12e50b1d709423daefbe972ffd3d34837c39a248634f5f52: Status 404 returned error can't find the container with id 4addc61a5efad3af12e50b1d709423daefbe972ffd3d34837c39a248634f5f52
Oct 31 13:15:29.154331 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.154290    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:15:29.154494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.154385    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:15:29.154494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.154432    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:15:29.176539 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.176426    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:29.176539 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.176451    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:29.176539 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.176464    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67r57\" (UniqueName: \"kubernetes.io/projected/c65d5f1f-befb-4c32-8f0e-0a09f4915b53-kube-api-access-67r57\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:29.183028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.182996    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6" event={"ID":"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4","Type":"ContainerStarted","Data":"ba06146b6946b33a50cd85de35d6ef9dda136a7c9ce019e772eedc84e64b6069"}
Oct 31 13:15:29.184717 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.184678    5787 generic.go:356] "Generic (PLEG): container finished" podID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerID="33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962" exitCode=0
Oct 31 13:15:29.184805 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.184762    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"
Oct 31 13:15:29.184860 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.184759    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" event={"ID":"04a8e1d5-8801-435a-a98e-e4ae80cf10a6","Type":"ContainerDied","Data":"33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962"}
Oct 31 13:15:29.184913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.184875    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m" event={"ID":"04a8e1d5-8801-435a-a98e-e4ae80cf10a6","Type":"ContainerDied","Data":"c8fdb596a0752aed936c4426d22a7c6309f6d545d370f2920113af590c5d5138"}
Oct 31 13:15:29.184913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.184900    5787 scope.go:122] "RemoveContainer" containerID="33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962"
Oct 31 13:15:29.186559 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.186536    5787 generic.go:356] "Generic (PLEG): container finished" podID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerID="13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb" exitCode=0
Oct 31 13:15:29.186669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.186590    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"
Oct 31 13:15:29.186669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.186635    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" event={"ID":"bf8cdef9-566f-443d-9949-3a8e9947fd4e","Type":"ContainerDied","Data":"13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb"}
Oct 31 13:15:29.186869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.186673    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n" event={"ID":"bf8cdef9-566f-443d-9949-3a8e9947fd4e","Type":"ContainerDied","Data":"20dcc84f7eb3cf1b6e82dfd6d7e3cbdc563bb67a89844d86f3b70f414e270306"}
Oct 31 13:15:29.193311 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.193287    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" event={"ID":"27fe87e5-d534-4f37-8260-5d5b415df0ff","Type":"ContainerStarted","Data":"4addc61a5efad3af12e50b1d709423daefbe972ffd3d34837c39a248634f5f52"}
Oct 31 13:15:29.195292 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.195268    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2" event={"ID":"6460e2c0-e9ba-4943-aee0-c06f74ede22d","Type":"ContainerStarted","Data":"4d08c400cc17ccda0a1c4112ea308734f990051cdd3d636fe0b24257c71f157b"}
Oct 31 13:15:29.196609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.196586    5787 generic.go:356] "Generic (PLEG): container finished" podID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerID="2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0" exitCode=0
Oct 31 13:15:29.196696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.196638    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"
Oct 31 13:15:29.196696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.196665    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" event={"ID":"c43584fa-29e8-4ab5-ab4e-366ed24bade8","Type":"ContainerDied","Data":"2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0"}
Oct 31 13:15:29.196805 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.196716    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" event={"ID":"c43584fa-29e8-4ab5-ab4e-366ed24bade8","Type":"ContainerDied","Data":"b3cdf3d13698c5174fc1396a3f16f608ed618f0cd2bb558919eede941a3ed53c"}
Oct 31 13:15:29.198338 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.198316    5787 generic.go:356] "Generic (PLEG): container finished" podID="5817ea86-18de-470f-b84e-8ff152d98f61" containerID="d2af4d61a4bd1ecbaa7cd5d8a3a70c9ec848bbcba3467d5184eda1a221a1405f" exitCode=0
Oct 31 13:15:29.198421 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.198394    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h" event={"ID":"5817ea86-18de-470f-b84e-8ff152d98f61","Type":"ContainerDied","Data":"d2af4d61a4bd1ecbaa7cd5d8a3a70c9ec848bbcba3467d5184eda1a221a1405f"}
Oct 31 13:15:29.199831 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.199806    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"
Oct 31 13:15:29.199831 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.199817    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5" event={"ID":"d366feca-7052-4618-8a80-b929afe65e16","Type":"ContainerDied","Data":"2392824da8406056f27fe2a19a60452ee0ec06fcd0735119cb240fc672719982"}
Oct 31 13:15:29.201593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.201575    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr" event={"ID":"efde1a18-f86c-4247-bd41-9ac23d244500","Type":"ContainerStarted","Data":"f480bfba1603856c166411cfca2aec8783a95a00adfc6c300f376a45a8aef507"}
Oct 31 13:15:29.201690 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.201617    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr" event={"ID":"efde1a18-f86c-4247-bd41-9ac23d244500","Type":"ContainerStarted","Data":"ce590d597a818fe632983256bce6523ef29060b182c24abee6f963cba09f8d99"}
Oct 31 13:15:29.202884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.202864    5787 generic.go:356] "Generic (PLEG): container finished" podID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerID="8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7" exitCode=0
Oct 31 13:15:29.202963 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.202885    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"
Oct 31 13:15:29.202963 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.202935    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" event={"ID":"c65d5f1f-befb-4c32-8f0e-0a09f4915b53","Type":"ContainerDied","Data":"8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7"}
Oct 31 13:15:29.202963 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.202962    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2" event={"ID":"c65d5f1f-befb-4c32-8f0e-0a09f4915b53","Type":"ContainerDied","Data":"d90baf6f74d681c91a9ad5b19a11a022f3df4093124ab2b3468ce9f37a891da2"}
Oct 31 13:15:29.203525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.203482    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"
Oct 31 13:15:29.217613 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.217591    5787 scope.go:122] "RemoveContainer" containerID="33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962"
Oct 31 13:15:29.217971 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:29.217948    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962\": container with ID starting with 33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962 not found: ID does not exist" containerID="33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962"
Oct 31 13:15:29.218034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.217981    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962"} err="failed to get container status \"33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962\": rpc error: code = NotFound desc = could not find container \"33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962\": container with ID starting with 33a9529c05ee7cb38d4d526e813f501bbcebb9078a5ac3e57a5cc3f1de673962 not found: ID does not exist"
Oct 31 13:15:29.218034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.218000    5787 scope.go:122] "RemoveContainer" containerID="13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb"
Oct 31 13:15:29.234635 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.234611    5787 scope.go:122] "RemoveContainer" containerID="13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb"
Oct 31 13:15:29.235094 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:29.235069    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb\": container with ID starting with 13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb not found: ID does not exist" containerID="13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb"
Oct 31 13:15:29.235189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.235103    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb"} err="failed to get container status \"13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb\": rpc error: code = NotFound desc = could not find container \"13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb\": container with ID starting with 13cbf115c03b479e1e1653d7ff4280db8ef944aafd21190e7c22a090d79552cb not found: ID does not exist"
Oct 31 13:15:29.235189 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.235121    5787 scope.go:122] "RemoveContainer" containerID="2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0"
Oct 31 13:15:29.258219 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.258194    5787 scope.go:122] "RemoveContainer" containerID="2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0"
Oct 31 13:15:29.258637 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:29.258599    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0\": container with ID starting with 2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0 not found: ID does not exist" containerID="2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0"
Oct 31 13:15:29.258745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.258653    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0"} err="failed to get container status \"2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0\": rpc error: code = NotFound desc = could not find container \"2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0\": container with ID starting with 2faed53c6a728df8b086a1938acba7259f131bbce82c251a7a57cbb0b955c0f0 not found: ID does not exist"
Oct 31 13:15:29.258745 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.258677    5787 scope.go:122] "RemoveContainer" containerID="699973dca7537958c1cf523e6274bde1aa94bccac02682d9644f47942ceba0a7"
Oct 31 13:15:29.259769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.259725    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2" podStartSLOduration=9.259692828 podStartE2EDuration="9.259692828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:29.223900477 +0000 UTC m=+908.639842318" watchObservedRunningTime="2026-10-31 13:15:29.259692828 +0000 UTC m=+908.675634680"
Oct 31 13:15:29.299915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.299787    5787 scope.go:122] "RemoveContainer" containerID="8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7"
Oct 31 13:15:29.302199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.302172    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"]
Oct 31 13:15:29.311770 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.311031    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/distinct-w5-f9e24f-kserve-6cb89b7c48-qcb7m"]
Oct 31 13:15:29.315548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.315524    5787 scope.go:122] "RemoveContainer" containerID="8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7"
Oct 31 13:15:29.315998 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:29.315972    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7\": container with ID starting with 8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7 not found: ID does not exist" containerID="8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7"
Oct 31 13:15:29.316060 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.316012    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7"} err="failed to get container status \"8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7\": rpc error: code = NotFound desc = could not find container \"8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7\": container with ID starting with 8ec30acc9cd6a1f5978cbd58ce53642f74d842c864f4b0c6b6ed7e9e4fe7e5f7 not found: ID does not exist"
Oct 31 13:15:29.318528 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.318471    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"]
Oct 31 13:15:29.323291 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.323263    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-w1-a2c006-kserve-76d4f94d4b-t2rg2"]
Oct 31 13:15:29.337674 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.337631    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:29.347905 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.347869    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"]
Oct 31 13:15:29.355756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.355720    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/simulator-w5-f9e24f-kserve-684b9b49bd-rp99v"]
Oct 31 13:15:29.368959 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.368929    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"]
Oct 31 13:15:29.388099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.387965    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-f9e24f/premium-w5-f9e24f-kserve-5d6b5cfd5-mbdr5"]
Oct 31 13:15:29.393994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.393962    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"]
Oct 31 13:15:29.398435 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.398404    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a2c006/distinct-2-w1-a2c006-kserve-d66648cdf-cnm4n"]
Oct 31 13:15:29.749972 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.749920    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"]
Oct 31 13:15:29.758169 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.758136    5787 cpu_manager.go:414] "containerMap: removing container" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main"
Oct 31 13:15:29.758637 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.758607    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main"
Oct 31 13:15:29.758950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.758928    5787 cpu_manager.go:414] "containerMap: removing container" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main"
Oct 31 13:15:29.760293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.760271    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main"
Oct 31 13:15:29.760439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.760426    5787 cpu_manager.go:414] "containerMap: removing container" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main"
Oct 31 13:15:29.760520 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.760509    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main"
Oct 31 13:15:29.760638 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.760625    5787 cpu_manager.go:414] "containerMap: removing container" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main"
Oct 31 13:15:29.760744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.760732    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main"
Oct 31 13:15:29.760845 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.760835    5787 cpu_manager.go:414] "containerMap: removing container" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main"
Oct 31 13:15:29.760933 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.760922    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main"
Oct 31 13:15:29.761206 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.761188    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" containerName="main"
Oct 31 13:15:29.761322 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.761311    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" containerName="main"
Oct 31 13:15:29.761415 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.761405    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" containerName="main"
Oct 31 13:15:29.761502 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.761492    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" containerName="main"
Oct 31 13:15:29.761590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.761580    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="d366feca-7052-4618-8a80-b929afe65e16" containerName="main"
Oct 31 13:15:29.773754 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.773480    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"]
Oct 31 13:15:29.774019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.773728    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:29.796808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.795537    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvgcn\" (UniqueName: \"kubernetes.io/projected/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-kube-api-access-xvgcn\") pod \"maas-api-revoke-sub-14d852eb-jb6c2\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:29.796808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.795727    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14d852eb-jb6c2\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:29.796808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.795916    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14d852eb-jb6c2\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:29.897465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.897372    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14d852eb-jb6c2\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:29.897649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.897502    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvgcn\" (UniqueName: \"kubernetes.io/projected/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-kube-api-access-xvgcn\") pod \"maas-api-revoke-sub-14d852eb-jb6c2\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:29.897649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.897581    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14d852eb-jb6c2\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:29.898166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.898145    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14d852eb-jb6c2\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:29.900073 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.900055    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14d852eb-jb6c2\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:29.910535 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:29.910508    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvgcn\" (UniqueName: \"kubernetes.io/projected/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-kube-api-access-xvgcn\") pod \"maas-api-revoke-sub-14d852eb-jb6c2\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:30.130967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.130929    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:30.277923 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.277842    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" event={"ID":"27fe87e5-d534-4f37-8260-5d5b415df0ff","Type":"ContainerStarted","Data":"4dd6ae5374e13bbc98fff885b4642abd98e04590a1b2d2b56c3fae2badc6b98e"}
Oct 31 13:15:30.280057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.280026    5787 generic.go:356] "Generic (PLEG): container finished" podID="6460e2c0-e9ba-4943-aee0-c06f74ede22d" containerID="4d08c400cc17ccda0a1c4112ea308734f990051cdd3d636fe0b24257c71f157b" exitCode=0
Oct 31 13:15:30.280203 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.280110    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2" event={"ID":"6460e2c0-e9ba-4943-aee0-c06f74ede22d","Type":"ContainerDied","Data":"4d08c400cc17ccda0a1c4112ea308734f990051cdd3d636fe0b24257c71f157b"}
Oct 31 13:15:30.283506 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.283481    5787 generic.go:356] "Generic (PLEG): container finished" podID="efde1a18-f86c-4247-bd41-9ac23d244500" containerID="f480bfba1603856c166411cfca2aec8783a95a00adfc6c300f376a45a8aef507" exitCode=0
Oct 31 13:15:30.283624 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.283579    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr" event={"ID":"efde1a18-f86c-4247-bd41-9ac23d244500","Type":"ContainerDied","Data":"f480bfba1603856c166411cfca2aec8783a95a00adfc6c300f376a45a8aef507"}
Oct 31 13:15:30.289100 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.288321    5787 generic.go:356] "Generic (PLEG): container finished" podID="9fc34dd9-ba4c-42f9-97d7-127a9958a9a4" containerID="3f85f7245083b90400f1e2d460a539419fc38b13709b7b9738e9770345e28536" exitCode=0
Oct 31 13:15:30.289100 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.288628    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6" event={"ID":"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4","Type":"ContainerDied","Data":"3f85f7245083b90400f1e2d460a539419fc38b13709b7b9738e9770345e28536"}
Oct 31 13:15:30.314027 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.313968    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" podStartSLOduration=8.313952303 podStartE2EDuration="8.313952303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:30.308188631 +0000 UTC m=+909.724130472" watchObservedRunningTime="2026-10-31 13:15:30.313952303 +0000 UTC m=+909.729894142"
Oct 31 13:15:30.466799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.456783    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"]
Oct 31 13:15:30.480771 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:30.480659    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2fa8ce_cfe1_4226_9825_25365b18e0bf.slice/crio-b89b20488f3d807d01378e2768999dcb612fd6cc4a1087c71bff1fe8c457e033 WatchSource:0}: Error finding container b89b20488f3d807d01378e2768999dcb612fd6cc4a1087c71bff1fe8c457e033: Status 404 returned error can't find the container with id b89b20488f3d807d01378e2768999dcb612fd6cc4a1087c71bff1fe8c457e033
Oct 31 13:15:30.624109 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.624074    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 13:15:30.624374 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.624342    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 13:15:30.624628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.624594    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:30.824115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.820617    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04a8e1d5-8801-435a-a98e-e4ae80cf10a6" path="/var/lib/kubelet/pods/04a8e1d5-8801-435a-a98e-e4ae80cf10a6/volumes"
Oct 31 13:15:30.824115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.821487    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8cdef9-566f-443d-9949-3a8e9947fd4e" path="/var/lib/kubelet/pods/bf8cdef9-566f-443d-9949-3a8e9947fd4e/volumes"
Oct 31 13:15:30.824115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.822131    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c43584fa-29e8-4ab5-ab4e-366ed24bade8" path="/var/lib/kubelet/pods/c43584fa-29e8-4ab5-ab4e-366ed24bade8/volumes"
Oct 31 13:15:30.846882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.846844    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c65d5f1f-befb-4c32-8f0e-0a09f4915b53" path="/var/lib/kubelet/pods/c65d5f1f-befb-4c32-8f0e-0a09f4915b53/volumes"
Oct 31 13:15:30.847885 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.847861    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d366feca-7052-4618-8a80-b929afe65e16" path="/var/lib/kubelet/pods/d366feca-7052-4618-8a80-b929afe65e16/volumes"
Oct 31 13:15:30.861732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.848467    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"]
Oct 31 13:15:30.862043 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.862018    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-f9e24f-vtdsj"]
Oct 31 13:15:30.994941 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:30.994910    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:31.171679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.162347    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efde1a18-f86c-4247-bd41-9ac23d244500-kube-api-access-65kqb\" (UniqueName: \"kubernetes.io/projected/efde1a18-f86c-4247-bd41-9ac23d244500-kube-api-access-65kqb\") pod \"efde1a18-f86c-4247-bd41-9ac23d244500\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") "
Oct 31 13:15:31.171679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.162439    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-xks-ca\") pod \"efde1a18-f86c-4247-bd41-9ac23d244500\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") "
Oct 31 13:15:31.171679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.162798    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-openshift-ca\") pod \"efde1a18-f86c-4247-bd41-9ac23d244500\" (UID: \"efde1a18-f86c-4247-bd41-9ac23d244500\") "
Oct 31 13:15:31.302339 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.302298    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-openshift-ca" pod "efde1a18-f86c-4247-bd41-9ac23d244500" (UID: "efde1a18-f86c-4247-bd41-9ac23d244500"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:31.418776 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.418741    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:31.422989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.422881    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efde1a18-f86c-4247-bd41-9ac23d244500-kube-api-access-65kqb" pod "efde1a18-f86c-4247-bd41-9ac23d244500" (UID: "efde1a18-f86c-4247-bd41-9ac23d244500"). InnerVolumeSpecName "kube-api-access-65kqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:31.423253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.423236    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-xks-ca" pod "efde1a18-f86c-4247-bd41-9ac23d244500" (UID: "efde1a18-f86c-4247-bd41-9ac23d244500"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:31.490669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.490634    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:31.517913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.517867    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2" event={"ID":"0a2fa8ce-cfe1-4226-9825-25365b18e0bf","Type":"ContainerStarted","Data":"b89b20488f3d807d01378e2768999dcb612fd6cc4a1087c71bff1fe8c457e033"}
Oct 31 13:15:31.522593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.520441    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65kqb\" (UniqueName: \"kubernetes.io/projected/efde1a18-f86c-4247-bd41-9ac23d244500-kube-api-access-65kqb\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:31.522593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.520471    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efde1a18-f86c-4247-bd41-9ac23d244500-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:31.522593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.522285    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h" event={"ID":"5817ea86-18de-470f-b84e-8ff152d98f61","Type":"ContainerDied","Data":"cf8e04951f418280f87ecf3f2074327aa4d153016da96d0704229abe6052d5ff"}
Oct 31 13:15:31.522593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.522335    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf8e04951f418280f87ecf3f2074327aa4d153016da96d0704229abe6052d5ff"
Oct 31 13:15:31.522593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.522408    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"
Oct 31 13:15:31.527767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.524658    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"
Oct 31 13:15:31.527767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.524698    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr" event={"ID":"efde1a18-f86c-4247-bd41-9ac23d244500","Type":"ContainerDied","Data":"ce590d597a818fe632983256bce6523ef29060b182c24abee6f963cba09f8d99"}
Oct 31 13:15:31.527767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.524825    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce590d597a818fe632983256bce6523ef29060b182c24abee6f963cba09f8d99"
Oct 31 13:15:31.597473 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.597441    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 13:15:31.597696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.597490    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 13:15:31.621160 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.621128    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-openshift-ca\") pod \"5817ea86-18de-470f-b84e-8ff152d98f61\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") "
Oct 31 13:15:31.621552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.621519    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-openshift-ca" pod "5817ea86-18de-470f-b84e-8ff152d98f61" (UID: "5817ea86-18de-470f-b84e-8ff152d98f61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:31.622050 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.621730    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-xks-ca\") pod \"5817ea86-18de-470f-b84e-8ff152d98f61\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") "
Oct 31 13:15:31.622050 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.621772    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5817ea86-18de-470f-b84e-8ff152d98f61-kube-api-access-c54nw\" (UniqueName: \"kubernetes.io/projected/5817ea86-18de-470f-b84e-8ff152d98f61-kube-api-access-c54nw\") pod \"5817ea86-18de-470f-b84e-8ff152d98f61\" (UID: \"5817ea86-18de-470f-b84e-8ff152d98f61\") "
Oct 31 13:15:31.623165 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.623143    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:31.670820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.670396    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-xks-ca" pod "5817ea86-18de-470f-b84e-8ff152d98f61" (UID: "5817ea86-18de-470f-b84e-8ff152d98f61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:31.670820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.670414    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5817ea86-18de-470f-b84e-8ff152d98f61-kube-api-access-c54nw" pod "5817ea86-18de-470f-b84e-8ff152d98f61" (UID: "5817ea86-18de-470f-b84e-8ff152d98f61"). InnerVolumeSpecName "kube-api-access-c54nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:31.727114 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.727043    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5817ea86-18de-470f-b84e-8ff152d98f61-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:31.728861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.727144    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54nw\" (UniqueName: \"kubernetes.io/projected/5817ea86-18de-470f-b84e-8ff152d98f61-kube-api-access-c54nw\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:31.884613 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.884575    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:31.888111 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.888085    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:31.977730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.977131    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-kube-api-access-6j6h8\" (UniqueName: \"kubernetes.io/projected/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-kube-api-access-6j6h8\") pod \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") "
Oct 31 13:15:31.977730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.977185    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-xks-ca\") pod \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") "
Oct 31 13:15:31.977730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.977239    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-openshift-ca\") pod \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\" (UID: \"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4\") "
Oct 31 13:15:31.977730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.977395    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6460e2c0-e9ba-4943-aee0-c06f74ede22d-kube-api-access-6dbqb\" (UniqueName: \"kubernetes.io/projected/6460e2c0-e9ba-4943-aee0-c06f74ede22d-kube-api-access-6dbqb\") pod \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") "
Oct 31 13:15:31.977730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.977445    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-openshift-ca\") pod \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") "
Oct 31 13:15:31.977730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.977555    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-xks-ca\") pod \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\" (UID: \"6460e2c0-e9ba-4943-aee0-c06f74ede22d\") "
Oct 31 13:15:31.991731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.982209    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-openshift-ca" pod "9fc34dd9-ba4c-42f9-97d7-127a9958a9a4" (UID: "9fc34dd9-ba4c-42f9-97d7-127a9958a9a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:31.992623 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:31.992312    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-openshift-ca" pod "6460e2c0-e9ba-4943-aee0-c06f74ede22d" (UID: "6460e2c0-e9ba-4943-aee0-c06f74ede22d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:32.057962 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.050648    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-xks-ca" pod "6460e2c0-e9ba-4943-aee0-c06f74ede22d" (UID: "6460e2c0-e9ba-4943-aee0-c06f74ede22d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:32.057962 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.052480    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-xks-ca" pod "9fc34dd9-ba4c-42f9-97d7-127a9958a9a4" (UID: "9fc34dd9-ba4c-42f9-97d7-127a9958a9a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:32.088086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.081371    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:32.088086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.081409    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:32.088086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.081427    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:32.088086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.081441    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6460e2c0-e9ba-4943-aee0-c06f74ede22d-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:32.106744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.106365    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-kube-api-access-6j6h8" pod "9fc34dd9-ba4c-42f9-97d7-127a9958a9a4" (UID: "9fc34dd9-ba4c-42f9-97d7-127a9958a9a4"). InnerVolumeSpecName "kube-api-access-6j6h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:32.108644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.108609    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6460e2c0-e9ba-4943-aee0-c06f74ede22d-kube-api-access-6dbqb" pod "6460e2c0-e9ba-4943-aee0-c06f74ede22d" (UID: "6460e2c0-e9ba-4943-aee0-c06f74ede22d"). InnerVolumeSpecName "kube-api-access-6dbqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:32.183814 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.183776    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dbqb\" (UniqueName: \"kubernetes.io/projected/6460e2c0-e9ba-4943-aee0-c06f74ede22d-kube-api-access-6dbqb\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:32.183814 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.183813    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j6h8\" (UniqueName: \"kubernetes.io/projected/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4-kube-api-access-6j6h8\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:32.560755 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.560678    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"
Oct 31 13:15:32.560980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.560684    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2" event={"ID":"6460e2c0-e9ba-4943-aee0-c06f74ede22d","Type":"ContainerDied","Data":"8c969a17b894342d949c9191fc5cf2396561042224a10c136d665221fb5fa906"}
Oct 31 13:15:32.560980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.560917    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c969a17b894342d949c9191fc5cf2396561042224a10c136d665221fb5fa906"
Oct 31 13:15:32.563645 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.563615    5787 generic.go:356] "Generic (PLEG): container finished" podID="0a2fa8ce-cfe1-4226-9825-25365b18e0bf" containerID="c59e61d90bc230ed7980a792d100165f71c1a9b4fd1e6a81929686a514743d37" exitCode=0
Oct 31 13:15:32.563799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.563749    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2" event={"ID":"0a2fa8ce-cfe1-4226-9825-25365b18e0bf","Type":"ContainerDied","Data":"c59e61d90bc230ed7980a792d100165f71c1a9b4fd1e6a81929686a514743d37"}
Oct 31 13:15:32.565605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.565586    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"
Oct 31 13:15:32.565697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.565621    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6" event={"ID":"9fc34dd9-ba4c-42f9-97d7-127a9958a9a4","Type":"ContainerDied","Data":"ba06146b6946b33a50cd85de35d6ef9dda136a7c9ce019e772eedc84e64b6069"}
Oct 31 13:15:32.565697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.565666    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba06146b6946b33a50cd85de35d6ef9dda136a7c9ce019e772eedc84e64b6069"
Oct 31 13:15:32.596620 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.596593    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 13:15:32.596774 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.596672    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 13:15:32.680117 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.680083    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"]
Oct 31 13:15:32.711698 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:32.711661    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ea089d-9578-4ea5-9361-0d0452da506d" path="/var/lib/kubelet/pods/78ea089d-9578-4ea5-9361-0d0452da506d/volumes"
Oct 31 13:15:33.367583 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.367538    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"]
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368335    5787 cpu_manager.go:414] "containerMap: removing container" podUID="efde1a18-f86c-4247-bd41-9ac23d244500" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368356    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efde1a18-f86c-4247-bd41-9ac23d244500" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368383    5787 cpu_manager.go:414] "containerMap: removing container" podUID="5817ea86-18de-470f-b84e-8ff152d98f61" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368390    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5817ea86-18de-470f-b84e-8ff152d98f61" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368417    5787 cpu_manager.go:414] "containerMap: removing container" podUID="9fc34dd9-ba4c-42f9-97d7-127a9958a9a4" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368425    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fc34dd9-ba4c-42f9-97d7-127a9958a9a4" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368438    5787 cpu_manager.go:414] "containerMap: removing container" podUID="6460e2c0-e9ba-4943-aee0-c06f74ede22d" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368444    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6460e2c0-e9ba-4943-aee0-c06f74ede22d" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368596    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="5817ea86-18de-470f-b84e-8ff152d98f61" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368608    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fc34dd9-ba4c-42f9-97d7-127a9958a9a4" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368615    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="6460e2c0-e9ba-4943-aee0-c06f74ede22d" containerName="revoke-keys"
Oct 31 13:15:33.368663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.368626    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="efde1a18-f86c-4247-bd41-9ac23d244500" containerName="revoke-keys"
Oct 31 13:15:33.374592 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.374522    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.374814 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.374545    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"]
Oct 31 13:15:33.417157 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.417126    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb7c570-vjg47\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.417331 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.417182    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb7c570-vjg47\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.417331 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.417198    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmgxx\" (UniqueName: \"kubernetes.io/projected/582d711c-00e3-4c13-922d-ed2e924a7eae-kube-api-access-hmgxx\") pod \"maas-api-revoke-sub-2eb7c570-vjg47\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.519285 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.519248    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb7c570-vjg47\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.519474 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.519310    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb7c570-vjg47\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.519474 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.519329    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmgxx\" (UniqueName: \"kubernetes.io/projected/582d711c-00e3-4c13-922d-ed2e924a7eae-kube-api-access-hmgxx\") pod \"maas-api-revoke-sub-2eb7c570-vjg47\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.520064 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.520037    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2eb7c570-vjg47\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.522183 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.522162    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2eb7c570-vjg47\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.533139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.533115    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmgxx\" (UniqueName: \"kubernetes.io/projected/582d711c-00e3-4c13-922d-ed2e924a7eae-kube-api-access-hmgxx\") pod \"maas-api-revoke-sub-2eb7c570-vjg47\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.596638 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.596605    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 13:15:33.596801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.596662    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 13:15:33.690722 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.690674    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:33.838009 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.833750    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:33.993784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.927435    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-kube-api-access-xvgcn\" (UniqueName: \"kubernetes.io/projected/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-kube-api-access-xvgcn\") pod \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") "
Oct 31 13:15:33.993784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.927549    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-openshift-ca\") pod \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") "
Oct 31 13:15:33.993784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.927621    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-xks-ca\") pod \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\" (UID: \"0a2fa8ce-cfe1-4226-9825-25365b18e0bf\") "
Oct 31 13:15:33.993784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.957391    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-openshift-ca" pod "0a2fa8ce-cfe1-4226-9825-25365b18e0bf" (UID: "0a2fa8ce-cfe1-4226-9825-25365b18e0bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:33.993784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:33.987957    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-kube-api-access-xvgcn" pod "0a2fa8ce-cfe1-4226-9825-25365b18e0bf" (UID: "0a2fa8ce-cfe1-4226-9825-25365b18e0bf"). InnerVolumeSpecName "kube-api-access-xvgcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:34.004930 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.004869    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-xks-ca" pod "0a2fa8ce-cfe1-4226-9825-25365b18e0bf" (UID: "0a2fa8ce-cfe1-4226-9825-25365b18e0bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:34.029764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.029723    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvgcn\" (UniqueName: \"kubernetes.io/projected/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-kube-api-access-xvgcn\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:34.029764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.029768    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:34.029961 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.029785    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a2fa8ce-cfe1-4226-9825-25365b18e0bf-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:34.256787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.255064    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"]
Oct 31 13:15:34.593779 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.593744    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2" event={"ID":"0a2fa8ce-cfe1-4226-9825-25365b18e0bf","Type":"ContainerDied","Data":"b89b20488f3d807d01378e2768999dcb612fd6cc4a1087c71bff1fe8c457e033"}
Oct 31 13:15:34.593779 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.593765    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"
Oct 31 13:15:34.594290 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.593802    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89b20488f3d807d01378e2768999dcb612fd6cc4a1087c71bff1fe8c457e033"
Oct 31 13:15:34.595796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.595767    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47" event={"ID":"582d711c-00e3-4c13-922d-ed2e924a7eae","Type":"ContainerStarted","Data":"606e201fbeb1e6ffc634ba693d66dccdb030d56282073552de6cb071a609c2d5"}
Oct 31 13:15:34.597004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.596978    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 13:15:34.597170 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:34.597031    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 13:15:35.611585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:35.608841    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 13:15:35.611585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:35.608931    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 13:15:35.686759 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:35.669272    5787 generic.go:356] "Generic (PLEG): container finished" podID="582d711c-00e3-4c13-922d-ed2e924a7eae" containerID="ac68659275efb474db7b100d8b777aa60195f873d5c4435a7279172ad00ee68a" exitCode=0
Oct 31 13:15:35.686759 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:35.669538    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47" event={"ID":"582d711c-00e3-4c13-922d-ed2e924a7eae","Type":"ContainerDied","Data":"ac68659275efb474db7b100d8b777aa60195f873d5c4435a7279172ad00ee68a"}
Oct 31 13:15:36.597051 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.597017    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body=
Oct 31 13:15:36.597214 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.597073    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused"
Oct 31 13:15:36.834616 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.834586    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:36.933463 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.933374    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-xks-ca\") pod \"582d711c-00e3-4c13-922d-ed2e924a7eae\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") "
Oct 31 13:15:36.933626 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.933613    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-openshift-ca\") pod \"582d711c-00e3-4c13-922d-ed2e924a7eae\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") "
Oct 31 13:15:36.933674 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.933644    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582d711c-00e3-4c13-922d-ed2e924a7eae-kube-api-access-hmgxx\" (UniqueName: \"kubernetes.io/projected/582d711c-00e3-4c13-922d-ed2e924a7eae-kube-api-access-hmgxx\") pod \"582d711c-00e3-4c13-922d-ed2e924a7eae\" (UID: \"582d711c-00e3-4c13-922d-ed2e924a7eae\") "
Oct 31 13:15:36.933989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.933952    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-openshift-ca" pod "582d711c-00e3-4c13-922d-ed2e924a7eae" (UID: "582d711c-00e3-4c13-922d-ed2e924a7eae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:36.934133 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.934123    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:36.935768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.935745    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-xks-ca" pod "582d711c-00e3-4c13-922d-ed2e924a7eae" (UID: "582d711c-00e3-4c13-922d-ed2e924a7eae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:36.936089 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:36.936069    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582d711c-00e3-4c13-922d-ed2e924a7eae-kube-api-access-hmgxx" pod "582d711c-00e3-4c13-922d-ed2e924a7eae" (UID: "582d711c-00e3-4c13-922d-ed2e924a7eae"). InnerVolumeSpecName "kube-api-access-hmgxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:37.036217 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:37.036177    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmgxx\" (UniqueName: \"kubernetes.io/projected/582d711c-00e3-4c13-922d-ed2e924a7eae-kube-api-access-hmgxx\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:37.036217 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:37.036218    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/582d711c-00e3-4c13-922d-ed2e924a7eae-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:37.601113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:37.601082    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:37.601614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:37.601577    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:37.602300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:37.602279    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:15:37.689636 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:37.689605    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"
Oct 31 13:15:37.689908 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:37.689608    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47" event={"ID":"582d711c-00e3-4c13-922d-ed2e924a7eae","Type":"ContainerDied","Data":"606e201fbeb1e6ffc634ba693d66dccdb030d56282073552de6cb071a609c2d5"}
Oct 31 13:15:37.690050 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:37.689978    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="606e201fbeb1e6ffc634ba693d66dccdb030d56282073552de6cb071a609c2d5"
Oct 31 13:15:39.623816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:39.623776    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"]
Oct 31 13:15:39.624289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:39.624040    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy" containerID="cri-o://b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8" gracePeriod=30
Oct 31 13:15:39.639224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:39.639192    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"]
Oct 31 13:15:39.639722 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:39.639680    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy" containerID="cri-o://d96380fb7e94a3a3cae82661f8384687866d8dfbdc6349fc055241f10bd2d939" gracePeriod=30
Oct 31 13:15:39.645873 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:39.645496    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"]
Oct 31 13:15:39.646007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:39.645967    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" containerID="cri-o://7c722275a5ad1c43ead8c1809c21b33b8203245a13f4815e8d2bda7232e47263" gracePeriod=30
Oct 31 13:15:39.653697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:39.653599    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"]
Oct 31 13:15:39.654523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:39.653854    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy" containerID="cri-o://a572f7af851bbb136ad598d533d7f2f8972207d9d54097132cb076a471a639ac" gracePeriod=30
Oct 31 13:15:40.671287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.671252    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"]
Oct 31 13:15:40.671927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.671910    5787 cpu_manager.go:414] "containerMap: removing container" podUID="582d711c-00e3-4c13-922d-ed2e924a7eae" containerName="revoke-keys"
Oct 31 13:15:40.671989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.671928    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582d711c-00e3-4c13-922d-ed2e924a7eae" containerName="revoke-keys"
Oct 31 13:15:40.671989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.671970    5787 cpu_manager.go:414] "containerMap: removing container" podUID="0a2fa8ce-cfe1-4226-9825-25365b18e0bf" containerName="revoke-keys"
Oct 31 13:15:40.671989 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.671975    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a2fa8ce-cfe1-4226-9825-25365b18e0bf" containerName="revoke-keys"
Oct 31 13:15:40.672097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.672074    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a2fa8ce-cfe1-4226-9825-25365b18e0bf" containerName="revoke-keys"
Oct 31 13:15:40.672097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.672086    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="582d711c-00e3-4c13-922d-ed2e924a7eae" containerName="revoke-keys"
Oct 31 13:15:40.676682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.676663    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.684752 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.684623    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:15:40.690816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.685990    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:15:40.690816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.686062    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:15:40.690816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.686107    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:15:40.690816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.690474    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"]
Oct 31 13:15:40.756330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.756297    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/ab5ad7fa-3594-4a7a-bc19-56642630e893-kube-api-access-b5k9g\") pod \"maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.756494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.756366    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.756494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.756461    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.857881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.857845    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/ab5ad7fa-3594-4a7a-bc19-56642630e893-kube-api-access-b5k9g\") pod \"maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.858077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.857960    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.858077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.858014    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.858639 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.858614    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.860666 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.860649    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.873168 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.873140    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/ab5ad7fa-3594-4a7a-bc19-56642630e893-kube-api-access-b5k9g\") pod \"maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:40.877460 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.877437    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"]
Oct 31 13:15:40.883055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.883033    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:40.891969 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.891945    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"]
Oct 31 13:15:40.959853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.959810    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7544\" (UniqueName: \"kubernetes.io/projected/85edd540-45e5-473b-b1a5-aa38d139b413-kube-api-access-w7544\") pod \"maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:40.960198 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.960172    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:40.960325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:40.960304    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:41.010761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.007852    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:41.062229 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.062190    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:41.062604 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.062578    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7544\" (UniqueName: \"kubernetes.io/projected/85edd540-45e5-473b-b1a5-aa38d139b413-kube-api-access-w7544\") pod \"maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:41.062725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.062675    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:41.063045 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.063018    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:41.067680 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.067637    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:41.085534 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.085507    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7544\" (UniqueName: \"kubernetes.io/projected/85edd540-45e5-473b-b1a5-aa38d139b413-kube-api-access-w7544\") pod \"maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:41.202231 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.202197    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:41.433039 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.433004    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"]
Oct 31 13:15:41.433479 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:41.433442    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab5ad7fa_3594_4a7a_bc19_56642630e893.slice/crio-3def88486e24657b12de5ed94d52d1c321d1962652b486ec937e5940dc1b79a2 WatchSource:0}: Error finding container 3def88486e24657b12de5ed94d52d1c321d1962652b486ec937e5940dc1b79a2: Status 404 returned error can't find the container with id 3def88486e24657b12de5ed94d52d1c321d1962652b486ec937e5940dc1b79a2
Oct 31 13:15:41.538303 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:41.538261    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"]
Oct 31 13:15:41.541267 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:41.541016    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85edd540_45e5_473b_b1a5_aa38d139b413.slice/crio-db77d8361b5992d79440826c16fa7b0e7c2aca2e40478641986431b60370f09c WatchSource:0}: Error finding container db77d8361b5992d79440826c16fa7b0e7c2aca2e40478641986431b60370f09c: Status 404 returned error can't find the container with id db77d8361b5992d79440826c16fa7b0e7c2aca2e40478641986431b60370f09c
Oct 31 13:15:42.046092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.045971    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l" event={"ID":"ab5ad7fa-3594-4a7a-bc19-56642630e893","Type":"ContainerStarted","Data":"e5b8e967fcb8846e69bddbfd2775c65b2ebce65eff397e478e1ac5cd120f0b75"}
Oct 31 13:15:42.046092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.046050    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l" event={"ID":"ab5ad7fa-3594-4a7a-bc19-56642630e893","Type":"ContainerStarted","Data":"3def88486e24657b12de5ed94d52d1c321d1962652b486ec937e5940dc1b79a2"}
Oct 31 13:15:42.048785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.048747    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s" event={"ID":"85edd540-45e5-473b-b1a5-aa38d139b413","Type":"ContainerStarted","Data":"c71e02e55ef5d9c59e3d8ef1f96b212278afe2ab0412e599eb4308a89c37577d"}
Oct 31 13:15:42.048957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.048814    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s" event={"ID":"85edd540-45e5-473b-b1a5-aa38d139b413","Type":"ContainerStarted","Data":"db77d8361b5992d79440826c16fa7b0e7c2aca2e40478641986431b60370f09c"}
Oct 31 13:15:42.071099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.071044    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l" podStartSLOduration=2.071029231 podStartE2EDuration="2.071029231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:42.067671355 +0000 UTC m=+921.483613196" watchObservedRunningTime="2026-10-31 13:15:42.071029231 +0000 UTC m=+921.486971071"
Oct 31 13:15:42.095359 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.095283    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s" podStartSLOduration=2.095261135 podStartE2EDuration="2.095261135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:42.090280583 +0000 UTC m=+921.506222424" watchObservedRunningTime="2026-10-31 13:15:42.095261135 +0000 UTC m=+921.511202976"
Oct 31 13:15:42.667282 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.667242    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:15:42.667438 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.667314    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:15:42.874785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.873953    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"]
Oct 31 13:15:42.921382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.908023    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"]
Oct 31 13:15:42.921382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.908162    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:42.943377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.931420    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea26cbb-2clrl\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:42.943377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.931509    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7rx\" (UniqueName: \"kubernetes.io/projected/3a835cb0-8e7f-4ca7-943d-1813763f3a36-kube-api-access-2t7rx\") pod \"maas-api-revoke-sub-9ea26cbb-2clrl\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:42.943377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.931821    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea26cbb-2clrl\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:42.943377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.934068    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"]
Oct 31 13:15:42.947301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.946358    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:42.956876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:42.956836    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"]
Oct 31 13:15:43.033337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.033298    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea26cbb-2clrl\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:43.033337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.033342    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhn6b\" (UniqueName: \"kubernetes.io/projected/d013b999-e4c4-41df-9c4f-961babdabcac-kube-api-access-fhn6b\") pod \"maas-api-revoke-sub-524e5f43-k2chm\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.033581 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.033412    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7rx\" (UniqueName: \"kubernetes.io/projected/3a835cb0-8e7f-4ca7-943d-1813763f3a36-kube-api-access-2t7rx\") pod \"maas-api-revoke-sub-9ea26cbb-2clrl\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:43.033581 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.033483    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-524e5f43-k2chm\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.033660 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.033632    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-524e5f43-k2chm\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.033800 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.033785    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea26cbb-2clrl\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:43.034036 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.034015    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea26cbb-2clrl\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:43.036280 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.036250    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea26cbb-2clrl\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:43.047291 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.047258    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7rx\" (UniqueName: \"kubernetes.io/projected/3a835cb0-8e7f-4ca7-943d-1813763f3a36-kube-api-access-2t7rx\") pod \"maas-api-revoke-sub-9ea26cbb-2clrl\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:43.059746 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.059683    5787 generic.go:356] "Generic (PLEG): container finished" podID="ab5ad7fa-3594-4a7a-bc19-56642630e893" containerID="e5b8e967fcb8846e69bddbfd2775c65b2ebce65eff397e478e1ac5cd120f0b75" exitCode=0
Oct 31 13:15:43.059931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.059747    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l" event={"ID":"ab5ad7fa-3594-4a7a-bc19-56642630e893","Type":"ContainerDied","Data":"e5b8e967fcb8846e69bddbfd2775c65b2ebce65eff397e478e1ac5cd120f0b75"}
Oct 31 13:15:43.061806 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.061782    5787 generic.go:356] "Generic (PLEG): container finished" podID="85edd540-45e5-473b-b1a5-aa38d139b413" containerID="c71e02e55ef5d9c59e3d8ef1f96b212278afe2ab0412e599eb4308a89c37577d" exitCode=0
Oct 31 13:15:43.061911 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.061834    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s" event={"ID":"85edd540-45e5-473b-b1a5-aa38d139b413","Type":"ContainerDied","Data":"c71e02e55ef5d9c59e3d8ef1f96b212278afe2ab0412e599eb4308a89c37577d"}
Oct 31 13:15:43.135897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.135809    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-524e5f43-k2chm\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.136119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.136094    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhn6b\" (UniqueName: \"kubernetes.io/projected/d013b999-e4c4-41df-9c4f-961babdabcac-kube-api-access-fhn6b\") pod \"maas-api-revoke-sub-524e5f43-k2chm\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.136257 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.136148    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-524e5f43-k2chm\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.137371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.137162    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-524e5f43-k2chm\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.139654 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.139621    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-524e5f43-k2chm\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.154400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.154364    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhn6b\" (UniqueName: \"kubernetes.io/projected/d013b999-e4c4-41df-9c4f-961babdabcac-kube-api-access-fhn6b\") pod \"maas-api-revoke-sub-524e5f43-k2chm\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.196161 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.196124    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main" containerID="cri-o://8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d" gracePeriod=45
Oct 31 13:15:43.267328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.256625    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:43.267328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.267068    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:43.573363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.573334    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:15:43.651592 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.651523    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"]
Oct 31 13:15:43.666889 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.666695    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"]
Oct 31 13:15:43.672763 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:43.672726    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a835cb0_8e7f_4ca7_943d_1813763f3a36.slice/crio-167b7acfa4008f05e80a4f8ebb83ff2d3be723f4f4510436c921a8dadd3a02da WatchSource:0}: Error finding container 167b7acfa4008f05e80a4f8ebb83ff2d3be723f4f4510436c921a8dadd3a02da: Status 404 returned error can't find the container with id 167b7acfa4008f05e80a4f8ebb83ff2d3be723f4f4510436c921a8dadd3a02da
Oct 31 13:15:43.675317 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.675291    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-home\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-home\") pod \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") "
Oct 31 13:15:43.675451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.675422    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-model-cache\") pod \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") "
Oct 31 13:15:43.675510 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.675455    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-dshm\") pod \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") "
Oct 31 13:15:43.675567 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.675516    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tls-certs\") pod \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") "
Oct 31 13:15:43.675567 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.675560    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-kube-api-access-bqqvd\" (UniqueName: \"kubernetes.io/projected/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-kube-api-access-bqqvd\") pod \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") "
Oct 31 13:15:43.676234 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.676004    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-home" pod "7416b2b0-1a8a-4887-aec9-cc7592a5de9e" (UID: "7416b2b0-1a8a-4887-aec9-cc7592a5de9e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:43.676234 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.676166    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-model-cache" pod "7416b2b0-1a8a-4887-aec9-cc7592a5de9e" (UID: "7416b2b0-1a8a-4887-aec9-cc7592a5de9e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:43.676367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.676349    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tmp-dir\") pod \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\" (UID: \"7416b2b0-1a8a-4887-aec9-cc7592a5de9e\") "
Oct 31 13:15:43.676956 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.676935    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:43.677054 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.676959    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:43.677194 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.677175    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tmp-dir" pod "7416b2b0-1a8a-4887-aec9-cc7592a5de9e" (UID: "7416b2b0-1a8a-4887-aec9-cc7592a5de9e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:43.679162 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.679128    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tls-certs" pod "7416b2b0-1a8a-4887-aec9-cc7592a5de9e" (UID: "7416b2b0-1a8a-4887-aec9-cc7592a5de9e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:43.679904 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.679871    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-dshm" pod "7416b2b0-1a8a-4887-aec9-cc7592a5de9e" (UID: "7416b2b0-1a8a-4887-aec9-cc7592a5de9e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:43.680471 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.680449    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-kube-api-access-bqqvd" pod "7416b2b0-1a8a-4887-aec9-cc7592a5de9e" (UID: "7416b2b0-1a8a-4887-aec9-cc7592a5de9e"). InnerVolumeSpecName "kube-api-access-bqqvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:43.778853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.778820    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:43.778853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.778850    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:43.778853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.778860    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqqvd\" (UniqueName: \"kubernetes.io/projected/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-kube-api-access-bqqvd\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:43.779087 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:43.778874    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7416b2b0-1a8a-4887-aec9-cc7592a5de9e-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:44.075720 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.075675    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl" event={"ID":"3a835cb0-8e7f-4ca7-943d-1813763f3a36","Type":"ContainerStarted","Data":"167b7acfa4008f05e80a4f8ebb83ff2d3be723f4f4510436c921a8dadd3a02da"}
Oct 31 13:15:44.077722 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.077676    5787 generic.go:356] "Generic (PLEG): container finished" podID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerID="8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d" exitCode=0
Oct 31 13:15:44.077885 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.077857    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" event={"ID":"7416b2b0-1a8a-4887-aec9-cc7592a5de9e","Type":"ContainerDied","Data":"8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d"}
Oct 31 13:15:44.077969 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.077933    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z" event={"ID":"7416b2b0-1a8a-4887-aec9-cc7592a5de9e","Type":"ContainerDied","Data":"ad22bc0f83881b62e636dfddba92a4b41bfbb7bd864f83a892a898eb5c68ebc7"}
Oct 31 13:15:44.078753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.077958    5787 scope.go:122] "RemoveContainer" containerID="8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d"
Oct 31 13:15:44.078896 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.078044    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"
Oct 31 13:15:44.081304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.081277    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm" event={"ID":"d013b999-e4c4-41df-9c4f-961babdabcac","Type":"ContainerStarted","Data":"be9156e4c43e94d437f9134b57e2b23bcad7864e49ca85fb42c7076db2107e4a"}
Oct 31 13:15:44.081432 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.081329    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm" event={"ID":"d013b999-e4c4-41df-9c4f-961babdabcac","Type":"ContainerStarted","Data":"077e7d8c577af43b4b2e636893ed7d8a4ef8ba45966fce0756585e478ab6f8d4"}
Oct 31 13:15:44.143832 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.141351    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm" podStartSLOduration=2.141327952 podStartE2EDuration="2.141327952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:44.119244319 +0000 UTC m=+923.535186159" watchObservedRunningTime="2026-10-31 13:15:44.141327952 +0000 UTC m=+923.557269792"
Oct 31 13:15:44.215644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.205934    5787 scope.go:122] "RemoveContainer" containerID="8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d"
Oct 31 13:15:44.215644 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:44.206392    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d\": container with ID starting with 8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d not found: ID does not exist" containerID="8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d"
Oct 31 13:15:44.215644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.206433    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d"} err="failed to get container status \"8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d\": rpc error: code = NotFound desc = could not find container \"8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d\": container with ID starting with 8da45c20ff5d3371257eac350a6fe79307d1e7e2da6a1841473c8aa710107d6d not found: ID does not exist"
Oct 31 13:15:44.266934 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.253889    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"]
Oct 31 13:15:44.428979 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.379993    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-2a262a/simulator-w2-2a262a-kserve-6bcd8d9fd5-xmh8z"]
Oct 31 13:15:44.437591 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.437262    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"]
Oct 31 13:15:44.445306 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.445270    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-a2c006-fcjsm"]
Oct 31 13:15:44.915930 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:44.913547    5787 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a714d74_2bc2_4eb2_b3f4_5471047b8cf5.slice/crio-conmon-7c722275a5ad1c43ead8c1809c21b33b8203245a13f4815e8d2bda7232e47263.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a714d74_2bc2_4eb2_b3f4_5471047b8cf5.slice/crio-7c722275a5ad1c43ead8c1809c21b33b8203245a13f4815e8d2bda7232e47263.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321518c6_acce_489e_9cf5_67a29bafc5ad.slice/crio-conmon-a572f7af851bbb136ad598d533d7f2f8972207d9d54097132cb076a471a639ac.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e11ad3c_692f_4245_a6d8_68b4f945b3d1.slice/crio-conmon-d96380fb7e94a3a3cae82661f8384687866d8dfbdc6349fc055241f10bd2d939.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2436534_3062_4a6d_ba08_93e324b1ec85.slice/crio-conmon-b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd013b999_e4c4_41df_9c4f_961babdabcac.slice/crio-conmon-be9156e4c43e94d437f9134b57e2b23bcad7864e49ca85fb42c7076db2107e4a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321518c6_acce_489e_9cf5_67a29bafc5ad.slice/crio-a572f7af851bbb136ad598d533d7f2f8972207d9d54097132cb076a471a639ac.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:15:44.921133 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.918872    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" path="/var/lib/kubelet/pods/7416b2b0-1a8a-4887-aec9-cc7592a5de9e/volumes"
Oct 31 13:15:44.921133 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.919540    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b35b374-4ada-444e-8947-c1abf0bf5fe7" path="/var/lib/kubelet/pods/8b35b374-4ada-444e-8947-c1abf0bf5fe7/volumes"
Oct 31 13:15:44.939784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.939492    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 31 13:15:44.939784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:44.939558    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 31 13:15:45.036813 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.036421    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:45.183737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.180514    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85edd540-45e5-473b-b1a5-aa38d139b413-kube-api-access-w7544\" (UniqueName: \"kubernetes.io/projected/85edd540-45e5-473b-b1a5-aa38d139b413-kube-api-access-w7544\") pod \"85edd540-45e5-473b-b1a5-aa38d139b413\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") "
Oct 31 13:15:45.183737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.180596    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-openshift-ca\") pod \"85edd540-45e5-473b-b1a5-aa38d139b413\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") "
Oct 31 13:15:45.183737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.180732    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-xks-ca\") pod \"85edd540-45e5-473b-b1a5-aa38d139b413\" (UID: \"85edd540-45e5-473b-b1a5-aa38d139b413\") "
Oct 31 13:15:45.244933 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.213365    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-openshift-ca" pod "85edd540-45e5-473b-b1a5-aa38d139b413" (UID: "85edd540-45e5-473b-b1a5-aa38d139b413"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:45.299277 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.299237    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.303398 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.302537    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:45.363195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.332485    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-xks-ca" pod "85edd540-45e5-473b-b1a5-aa38d139b413" (UID: "85edd540-45e5-473b-b1a5-aa38d139b413"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:45.389544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.389500    5787 patch_prober.go:29] interesting pod/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Oct 31 13:15:45.389857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.389815    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Oct 31 13:15:45.415756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.415722    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab5ad7fa-3594-4a7a-bc19-56642630e893-kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/ab5ad7fa-3594-4a7a-bc19-56642630e893-kube-api-access-b5k9g\") pod \"ab5ad7fa-3594-4a7a-bc19-56642630e893\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") "
Oct 31 13:15:45.422023 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.421952    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-openshift-ca\") pod \"ab5ad7fa-3594-4a7a-bc19-56642630e893\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") "
Oct 31 13:15:45.422516 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.422491    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-xks-ca\") pod \"ab5ad7fa-3594-4a7a-bc19-56642630e893\" (UID: \"ab5ad7fa-3594-4a7a-bc19-56642630e893\") "
Oct 31 13:15:45.424596 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.424348    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85edd540-45e5-473b-b1a5-aa38d139b413-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.477270 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.466593    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl" event={"ID":"3a835cb0-8e7f-4ca7-943d-1813763f3a36","Type":"ContainerStarted","Data":"5038e3d66a70284e1fc4c718d8fa9e474b2e6e935c95001d48a1dce8c0c9c184"}
Oct 31 13:15:45.477863 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.477783    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l" event={"ID":"ab5ad7fa-3594-4a7a-bc19-56642630e893","Type":"ContainerDied","Data":"3def88486e24657b12de5ed94d52d1c321d1962652b486ec937e5940dc1b79a2"}
Oct 31 13:15:45.477863 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.477846    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3def88486e24657b12de5ed94d52d1c321d1962652b486ec937e5940dc1b79a2"
Oct 31 13:15:45.478085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.477961    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"
Oct 31 13:15:45.479259 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.478275    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85edd540-45e5-473b-b1a5-aa38d139b413-kube-api-access-w7544" pod "85edd540-45e5-473b-b1a5-aa38d139b413" (UID: "85edd540-45e5-473b-b1a5-aa38d139b413"). InnerVolumeSpecName "kube-api-access-w7544". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.481991 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.481952    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-openshift-ca" pod "ab5ad7fa-3594-4a7a-bc19-56642630e893" (UID: "ab5ad7fa-3594-4a7a-bc19-56642630e893"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:45.494997 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.493300    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-xks-ca" pod "ab5ad7fa-3594-4a7a-bc19-56642630e893" (UID: "ab5ad7fa-3594-4a7a-bc19-56642630e893"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:45.496943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.487938    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm" event={"ID":"d013b999-e4c4-41df-9c4f-961babdabcac","Type":"ContainerDied","Data":"be9156e4c43e94d437f9134b57e2b23bcad7864e49ca85fb42c7076db2107e4a"}
Oct 31 13:15:45.497516 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.497455    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab5ad7fa-3594-4a7a-bc19-56642630e893-kube-api-access-b5k9g" pod "ab5ad7fa-3594-4a7a-bc19-56642630e893" (UID: "ab5ad7fa-3594-4a7a-bc19-56642630e893"). InnerVolumeSpecName "kube-api-access-b5k9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.498414 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.487908    5787 generic.go:356] "Generic (PLEG): container finished" podID="d013b999-e4c4-41df-9c4f-961babdabcac" containerID="be9156e4c43e94d437f9134b57e2b23bcad7864e49ca85fb42c7076db2107e4a" exitCode=0
Oct 31 13:15:45.503374 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.500763    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"
Oct 31 13:15:45.505822 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.505682    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s" event={"ID":"85edd540-45e5-473b-b1a5-aa38d139b413","Type":"ContainerDied","Data":"db77d8361b5992d79440826c16fa7b0e7c2aca2e40478641986431b60370f09c"}
Oct 31 13:15:45.505822 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.505769    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db77d8361b5992d79440826c16fa7b0e7c2aca2e40478641986431b60370f09c"
Oct 31 13:15:45.548602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.541950    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"]
Oct 31 13:15:45.575768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.573631    5787 generic.go:356] "Generic (PLEG): container finished" podID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerID="7c722275a5ad1c43ead8c1809c21b33b8203245a13f4815e8d2bda7232e47263" exitCode=0
Oct 31 13:15:45.575768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.573786    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" event={"ID":"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5","Type":"ContainerDied","Data":"7c722275a5ad1c43ead8c1809c21b33b8203245a13f4815e8d2bda7232e47263"}
Oct 31 13:15:45.577013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.576903    5787 generic.go:356] "Generic (PLEG): container finished" podID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerID="a572f7af851bbb136ad598d533d7f2f8972207d9d54097132cb076a471a639ac" exitCode=0
Oct 31 13:15:45.577013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.577007    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" event={"ID":"321518c6-acce-489e-9cf5-67a29bafc5ad","Type":"ContainerDied","Data":"a572f7af851bbb136ad598d533d7f2f8972207d9d54097132cb076a471a639ac"}
Oct 31 13:15:45.579053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.579023    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5k9g\" (UniqueName: \"kubernetes.io/projected/ab5ad7fa-3594-4a7a-bc19-56642630e893-kube-api-access-b5k9g\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.579156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.579058    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7544\" (UniqueName: \"kubernetes.io/projected/85edd540-45e5-473b-b1a5-aa38d139b413-kube-api-access-w7544\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.579156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.579074    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.579156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.579087    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab5ad7fa-3594-4a7a-bc19-56642630e893-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.579909 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.579854    5787 generic.go:356] "Generic (PLEG): container finished" podID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerID="d96380fb7e94a3a3cae82661f8384687866d8dfbdc6349fc055241f10bd2d939" exitCode=0
Oct 31 13:15:45.580010 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.579934    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" event={"ID":"4e11ad3c-692f-4245-a6d8-68b4f945b3d1","Type":"ContainerDied","Data":"d96380fb7e94a3a3cae82661f8384687866d8dfbdc6349fc055241f10bd2d939"}
Oct 31 13:15:45.585678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.585654    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"]
Oct 31 13:15:45.592752 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.592376    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"]
Oct 31 13:15:45.605309 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.605247    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl" podStartSLOduration=3.605226282 podStartE2EDuration="3.605226282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:45.51592938 +0000 UTC m=+924.931871223" watchObservedRunningTime="2026-10-31 13:15:45.605226282 +0000 UTC m=+925.021168123"
Oct 31 13:15:45.665676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.662207    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"]
Oct 31 13:15:45.686779 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.686670    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:15:45.689089 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.687841    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"]
Oct 31 13:15:45.689089 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.688816    5787 cpu_manager.go:414] "containerMap: removing container" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy"
Oct 31 13:15:45.689089 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.688835    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689386    5787 cpu_manager.go:414] "containerMap: removing container" podUID="85edd540-45e5-473b-b1a5-aa38d139b413" containerName="revoke-keys"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689406    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85edd540-45e5-473b-b1a5-aa38d139b413" containerName="revoke-keys"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689446    5787 cpu_manager.go:414] "containerMap: removing container" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689454    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689475    5787 cpu_manager.go:414] "containerMap: removing container" podUID="ab5ad7fa-3594-4a7a-bc19-56642630e893" containerName="revoke-keys"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689482    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab5ad7fa-3594-4a7a-bc19-56642630e893" containerName="revoke-keys"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689651    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="85edd540-45e5-473b-b1a5-aa38d139b413" containerName="revoke-keys"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689666    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="7416b2b0-1a8a-4887-aec9-cc7592a5de9e" containerName="main"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689681    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab5ad7fa-3594-4a7a-bc19-56642630e893" containerName="revoke-keys"
Oct 31 13:15:45.692145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.689692    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" containerName="istio-proxy"
Oct 31 13:15:45.696904 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.696742    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:45.711592 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.711560    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"]
Oct 31 13:15:45.731521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.731494    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"]
Oct 31 13:15:45.760132 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.760058    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:15:45.792993 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.792963    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-envoy\") pod \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") "
Oct 31 13:15:45.793178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793035    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-kube-api-access-72d85\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-kube-api-access-72d85\") pod \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") "
Oct 31 13:15:45.793178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793079    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-kube-api-access-94pkd\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-kube-api-access-94pkd\") pod \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") "
Oct 31 13:15:45.793178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793105    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-credential-socket\") pod \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") "
Oct 31 13:15:45.793178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793155    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-podinfo\") pod \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") "
Oct 31 13:15:45.793419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793250    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-podinfo\") pod \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") "
Oct 31 13:15:45.793419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793283    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istiod-ca-cert\") pod \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") "
Oct 31 13:15:45.793419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793314    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-credential-socket\") pod \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") "
Oct 31 13:15:45.793419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793352    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istiod-ca-cert\") pod \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") "
Oct 31 13:15:45.793616 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793419    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-socket\") pod \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") "
Oct 31 13:15:45.793616 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793504    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-token\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-token\") pod \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") "
Oct 31 13:15:45.793616 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793592    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-certs\") pod \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") "
Oct 31 13:15:45.793809 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793636    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-socket\") pod \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") "
Oct 31 13:15:45.793809 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793688    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-certs\") pod \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") "
Oct 31 13:15:45.793809 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793740    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-data\") pod \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\" (UID: \"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5\") "
Oct 31 13:15:45.793809 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793770    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-token\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-token\") pod \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") "
Oct 31 13:15:45.794006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793815    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-data\") pod \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") "
Oct 31 13:15:45.794006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.793873    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-envoy\") pod \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\" (UID: \"4e11ad3c-692f-4245-a6d8-68b4f945b3d1\") "
Oct 31 13:15:45.794446 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.794422    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8gd\" (UniqueName: \"kubernetes.io/projected/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-kube-api-access-rb8gd\") pod \"maas-api-revoke-sub-7e7d1149-c65cf\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:45.794567 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.794484    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7d1149-c65cf\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:45.794567 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.794540    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7d1149-c65cf\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:45.795544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.795505    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:15:45.795949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.795911    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-data" pod "4e11ad3c-692f-4245-a6d8-68b4f945b3d1" (UID: "4e11ad3c-692f-4245-a6d8-68b4f945b3d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.796119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.796095    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-socket" pod "4e11ad3c-692f-4245-a6d8-68b4f945b3d1" (UID: "4e11ad3c-692f-4245-a6d8-68b4f945b3d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.796281 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.796262    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-certs" pod "4e11ad3c-692f-4245-a6d8-68b4f945b3d1" (UID: "4e11ad3c-692f-4245-a6d8-68b4f945b3d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.796453 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.796436    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-data" pod "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" (UID: "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.797747 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.797405    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istiod-ca-cert" pod "4e11ad3c-692f-4245-a6d8-68b4f945b3d1" (UID: "4e11ad3c-692f-4245-a6d8-68b4f945b3d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:45.798434 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.798406    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-certs" pod "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" (UID: "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.798982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.798892    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istiod-ca-cert" pod "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" (UID: "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:45.799166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.799145    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-credential-socket" pod "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" (UID: "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.799233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.799195    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-credential-socket" pod "4e11ad3c-692f-4245-a6d8-68b4f945b3d1" (UID: "4e11ad3c-692f-4245-a6d8-68b4f945b3d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.799287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.799233    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-envoy" pod "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" (UID: "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.799471 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.799444    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-socket" pod "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" (UID: "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.801739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.801646    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-token" pod "4e11ad3c-692f-4245-a6d8-68b4f945b3d1" (UID: "4e11ad3c-692f-4245-a6d8-68b4f945b3d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.802064 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.802033    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-kube-api-access-72d85" pod "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" (UID: "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5"). InnerVolumeSpecName "kube-api-access-72d85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.802152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.802088    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-envoy" pod "4e11ad3c-692f-4245-a6d8-68b4f945b3d1" (UID: "4e11ad3c-692f-4245-a6d8-68b4f945b3d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.802213 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.802168    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-podinfo" pod "4e11ad3c-692f-4245-a6d8-68b4f945b3d1" (UID: "4e11ad3c-692f-4245-a6d8-68b4f945b3d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:15:45.802629 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.802600    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-token" pod "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" (UID: "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.803471 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.803444    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-kube-api-access-94pkd" pod "4e11ad3c-692f-4245-a6d8-68b4f945b3d1" (UID: "4e11ad3c-692f-4245-a6d8-68b4f945b3d1"). InnerVolumeSpecName "kube-api-access-94pkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.803699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.803678    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-podinfo" pod "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" (UID: "1a714d74-2bc2-4eb2-b3f4-5471047b8cf5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:15:45.830838 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.830809    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:15:45.895168 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.895073    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-envoy\") pod \"e2436534-3062-4a6d-ba08-93e324b1ec85\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") "
Oct 31 13:15:45.896078 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.896045    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-token\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-token\") pod \"321518c6-acce-489e-9cf5-67a29bafc5ad\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") "
Oct 31 13:15:45.896460 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.896437    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-token\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-token\") pod \"e2436534-3062-4a6d-ba08-93e324b1ec85\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") "
Oct 31 13:15:45.896866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.896838    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2436534-3062-4a6d-ba08-93e324b1ec85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2436534-3062-4a6d-ba08-93e324b1ec85-istiod-ca-cert\") pod \"e2436534-3062-4a6d-ba08-93e324b1ec85\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") "
Oct 31 13:15:45.897430 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897395    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-data\") pod \"e2436534-3062-4a6d-ba08-93e324b1ec85\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") "
Oct 31 13:15:45.897546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897433    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-kube-api-access-9nzjt\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-kube-api-access-9nzjt\") pod \"321518c6-acce-489e-9cf5-67a29bafc5ad\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") "
Oct 31 13:15:45.897546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897478    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-socket\") pod \"321518c6-acce-489e-9cf5-67a29bafc5ad\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") "
Oct 31 13:15:45.897546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897506    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-envoy\") pod \"321518c6-acce-489e-9cf5-67a29bafc5ad\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") "
Oct 31 13:15:45.897546 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897532    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-kube-api-access-79kv4\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-kube-api-access-79kv4\") pod \"e2436534-3062-4a6d-ba08-93e324b1ec85\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") "
Oct 31 13:15:45.897857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897629    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-certs\") pod \"e2436534-3062-4a6d-ba08-93e324b1ec85\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") "
Oct 31 13:15:45.897857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897736    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-credential-socket\") pod \"e2436534-3062-4a6d-ba08-93e324b1ec85\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") "
Oct 31 13:15:45.897857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897780    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/321518c6-acce-489e-9cf5-67a29bafc5ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321518c6-acce-489e-9cf5-67a29bafc5ad-istiod-ca-cert\") pod \"321518c6-acce-489e-9cf5-67a29bafc5ad\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") "
Oct 31 13:15:45.897857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897822    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-data\") pod \"321518c6-acce-489e-9cf5-67a29bafc5ad\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") "
Oct 31 13:15:45.897857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897848    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-credential-socket\") pod \"321518c6-acce-489e-9cf5-67a29bafc5ad\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") "
Oct 31 13:15:45.898145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897873    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-socket\") pod \"e2436534-3062-4a6d-ba08-93e324b1ec85\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") "
Oct 31 13:15:45.898145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.898108    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-podinfo\") pod \"e2436534-3062-4a6d-ba08-93e324b1ec85\" (UID: \"e2436534-3062-4a6d-ba08-93e324b1ec85\") "
Oct 31 13:15:45.898231 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.898159    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-podinfo\") pod \"321518c6-acce-489e-9cf5-67a29bafc5ad\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") "
Oct 31 13:15:45.898231 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.898191    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-certs\") pod \"321518c6-acce-489e-9cf5-67a29bafc5ad\" (UID: \"321518c6-acce-489e-9cf5-67a29bafc5ad\") "
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.897298    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2436534-3062-4a6d-ba08-93e324b1ec85-istiod-ca-cert" pod "e2436534-3062-4a6d-ba08-93e324b1ec85" (UID: "e2436534-3062-4a6d-ba08-93e324b1ec85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.898672    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-credential-socket" pod "321518c6-acce-489e-9cf5-67a29bafc5ad" (UID: "321518c6-acce-489e-9cf5-67a29bafc5ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.898942    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8gd\" (UniqueName: \"kubernetes.io/projected/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-kube-api-access-rb8gd\") pod \"maas-api-revoke-sub-7e7d1149-c65cf\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899126    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7d1149-c65cf\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899222    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7d1149-c65cf\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899453    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899473    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899490    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72d85\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-kube-api-access-72d85\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899505    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94pkd\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-kube-api-access-94pkd\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899519    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899534    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899549    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899566    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899580    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899592    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899651    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899668    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2436534-3062-4a6d-ba08-93e324b1ec85-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899683    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899697    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.900103    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.900120    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.900134    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.900150    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.900164    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.900179    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e11ad3c-692f-4245-a6d8-68b4f945b3d1-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899219    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-envoy" pod "e2436534-3062-4a6d-ba08-93e324b1ec85" (UID: "e2436534-3062-4a6d-ba08-93e324b1ec85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.899672    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-socket" pod "e2436534-3062-4a6d-ba08-93e324b1ec85" (UID: "e2436534-3062-4a6d-ba08-93e324b1ec85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.900408    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-credential-socket" pod "e2436534-3062-4a6d-ba08-93e324b1ec85" (UID: "e2436534-3062-4a6d-ba08-93e324b1ec85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.900699    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/321518c6-acce-489e-9cf5-67a29bafc5ad-istiod-ca-cert" pod "321518c6-acce-489e-9cf5-67a29bafc5ad" (UID: "321518c6-acce-489e-9cf5-67a29bafc5ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.901008    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-data" pod "321518c6-acce-489e-9cf5-67a29bafc5ad" (UID: "321518c6-acce-489e-9cf5-67a29bafc5ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.901403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.901246    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-kube-api-access-9nzjt" pod "321518c6-acce-489e-9cf5-67a29bafc5ad" (UID: "321518c6-acce-489e-9cf5-67a29bafc5ad"). InnerVolumeSpecName "kube-api-access-9nzjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.905382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.901443    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-data" pod "e2436534-3062-4a6d-ba08-93e324b1ec85" (UID: "e2436534-3062-4a6d-ba08-93e324b1ec85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.905382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.901678    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-envoy" pod "321518c6-acce-489e-9cf5-67a29bafc5ad" (UID: "321518c6-acce-489e-9cf5-67a29bafc5ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.905382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.901920    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-socket" pod "321518c6-acce-489e-9cf5-67a29bafc5ad" (UID: "321518c6-acce-489e-9cf5-67a29bafc5ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.905382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.902108    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-certs" pod "321518c6-acce-489e-9cf5-67a29bafc5ad" (UID: "321518c6-acce-489e-9cf5-67a29bafc5ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.905382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.902262    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-certs" pod "e2436534-3062-4a6d-ba08-93e324b1ec85" (UID: "e2436534-3062-4a6d-ba08-93e324b1ec85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:45.905382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.903012    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e7d1149-c65cf\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:45.905382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.904153    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-token" pod "e2436534-3062-4a6d-ba08-93e324b1ec85" (UID: "e2436534-3062-4a6d-ba08-93e324b1ec85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.905382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.904248    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-podinfo" pod "321518c6-acce-489e-9cf5-67a29bafc5ad" (UID: "321518c6-acce-489e-9cf5-67a29bafc5ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:15:45.905382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.905068    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-token" pod "321518c6-acce-489e-9cf5-67a29bafc5ad" (UID: "321518c6-acce-489e-9cf5-67a29bafc5ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.905942 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.905895    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-podinfo" pod "e2436534-3062-4a6d-ba08-93e324b1ec85" (UID: "e2436534-3062-4a6d-ba08-93e324b1ec85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:15:45.906516 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.906490    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e7d1149-c65cf\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:45.906829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.906810    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-kube-api-access-79kv4" pod "e2436534-3062-4a6d-ba08-93e324b1ec85" (UID: "e2436534-3062-4a6d-ba08-93e324b1ec85"). InnerVolumeSpecName "kube-api-access-79kv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:45.914344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:45.914319    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8gd\" (UniqueName: \"kubernetes.io/projected/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-kube-api-access-rb8gd\") pod \"maas-api-revoke-sub-7e7d1149-c65cf\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:46.002057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002018    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002051    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/321518c6-acce-489e-9cf5-67a29bafc5ad-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002062    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002071    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002080    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002088    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002097    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002105    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002113    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002120    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002129    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002138    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nzjt\" (UniqueName: \"kubernetes.io/projected/321518c6-acce-489e-9cf5-67a29bafc5ad-kube-api-access-9nzjt\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002147    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002157    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/321518c6-acce-489e-9cf5-67a29bafc5ad-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002181    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79kv4\" (UniqueName: \"kubernetes.io/projected/e2436534-3062-4a6d-ba08-93e324b1ec85-kube-api-access-79kv4\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.002267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.002189    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2436534-3062-4a6d-ba08-93e324b1ec85-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.027458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.027162    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:46.444761 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:46.444731    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6aa5d4_ed8f_4e81_a067_fc98f32e8a1e.slice/crio-94f10d279e0f07d30b6dd08d90a79dfe57725181186695e0596c066d280fbaf8 WatchSource:0}: Error finding container 94f10d279e0f07d30b6dd08d90a79dfe57725181186695e0596c066d280fbaf8: Status 404 returned error can't find the container with id 94f10d279e0f07d30b6dd08d90a79dfe57725181186695e0596c066d280fbaf8
Oct 31 13:15:46.446572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.446543    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"]
Oct 31 13:15:46.590547 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.590510    5787 generic.go:356] "Generic (PLEG): container finished" podID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerID="b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8" exitCode=0
Oct 31 13:15:46.590760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.590570    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"
Oct 31 13:15:46.590760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.590597    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" event={"ID":"e2436534-3062-4a6d-ba08-93e324b1ec85","Type":"ContainerDied","Data":"b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8"}
Oct 31 13:15:46.590760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.590651    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l" event={"ID":"e2436534-3062-4a6d-ba08-93e324b1ec85","Type":"ContainerDied","Data":"a75583a0bc0c4210524984178e97c03105a3178b18e891e0509c40f6eb1c3bc0"}
Oct 31 13:15:46.590760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.590672    5787 scope.go:122] "RemoveContainer" containerID="b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8"
Oct 31 13:15:46.592367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.592337    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"
Oct 31 13:15:46.593080 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.592344    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd" event={"ID":"1a714d74-2bc2-4eb2-b3f4-5471047b8cf5","Type":"ContainerDied","Data":"0ffbf2cca149488255ac77f7833925e23ec14d519beeb873fee4c30055bc4dcb"}
Oct 31 13:15:46.594182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.594155    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf" event={"ID":"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e","Type":"ContainerStarted","Data":"94f10d279e0f07d30b6dd08d90a79dfe57725181186695e0596c066d280fbaf8"}
Oct 31 13:15:46.595869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.595845    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf" event={"ID":"321518c6-acce-489e-9cf5-67a29bafc5ad","Type":"ContainerDied","Data":"290b481ab15ac6bac64e8eac2f16ae1a6dbbc5e9a5f4278f517d7146dc4b7394"}
Oct 31 13:15:46.595988 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.595893    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"
Oct 31 13:15:46.597451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.597427    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c" event={"ID":"4e11ad3c-692f-4245-a6d8-68b4f945b3d1","Type":"ContainerDied","Data":"df76726b436f4e81bf0d94c3c68b6830aadd1c1473e405b398ad04aaa79aca69"}
Oct 31 13:15:46.597557 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.597537    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"
Oct 31 13:15:46.601962 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.601938    5787 generic.go:356] "Generic (PLEG): container finished" podID="3a835cb0-8e7f-4ca7-943d-1813763f3a36" containerID="5038e3d66a70284e1fc4c718d8fa9e474b2e6e935c95001d48a1dce8c0c9c184" exitCode=0
Oct 31 13:15:46.602258 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.601982    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl" event={"ID":"3a835cb0-8e7f-4ca7-943d-1813763f3a36","Type":"ContainerDied","Data":"5038e3d66a70284e1fc4c718d8fa9e474b2e6e935c95001d48a1dce8c0c9c184"}
Oct 31 13:15:46.609241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.609155    5787 scope.go:122] "RemoveContainer" containerID="b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8"
Oct 31 13:15:46.609778 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:46.609752    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8\": container with ID starting with b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8 not found: ID does not exist" containerID="b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8"
Oct 31 13:15:46.609871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.609788    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8"} err="failed to get container status \"b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8\": rpc error: code = NotFound desc = could not find container \"b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8\": container with ID starting with b156a5e32fe25e08a68eb8d6d393a78d93bd05c0f187066e12e96aec036b6ac8 not found: ID does not exist"
Oct 31 13:15:46.609871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.609813    5787 scope.go:122] "RemoveContainer" containerID="7c722275a5ad1c43ead8c1809c21b33b8203245a13f4815e8d2bda7232e47263"
Oct 31 13:15:46.623132 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.623068    5787 scope.go:122] "RemoveContainer" containerID="a572f7af851bbb136ad598d533d7f2f8972207d9d54097132cb076a471a639ac"
Oct 31 13:15:46.634837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.634815    5787 scope.go:122] "RemoveContainer" containerID="d96380fb7e94a3a3cae82661f8384687866d8dfbdc6349fc055241f10bd2d939"
Oct 31 13:15:46.661937 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.661906    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"]
Oct 31 13:15:46.666929 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.666900    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-84bdf"]
Oct 31 13:15:46.695904 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.695805    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"]
Oct 31 13:15:46.730096 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.730057    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" path="/var/lib/kubelet/pods/321518c6-acce-489e-9cf5-67a29bafc5ad/volumes"
Oct 31 13:15:46.730604 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.730581    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-r2t9c"]
Oct 31 13:15:46.730688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.730636    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"]
Oct 31 13:15:46.735373 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.735342    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-hqbsd"]
Oct 31 13:15:46.743979 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.743949    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"]
Oct 31 13:15:46.752012 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.751984    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-f9e24f-openshift-default-f747b9cbc-pch2l"]
Oct 31 13:15:46.755273 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.755250    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:46.831397 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.831361    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-xks-ca\") pod \"d013b999-e4c4-41df-9c4f-961babdabcac\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") "
Oct 31 13:15:46.831585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.831445    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d013b999-e4c4-41df-9c4f-961babdabcac-kube-api-access-fhn6b\" (UniqueName: \"kubernetes.io/projected/d013b999-e4c4-41df-9c4f-961babdabcac-kube-api-access-fhn6b\") pod \"d013b999-e4c4-41df-9c4f-961babdabcac\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") "
Oct 31 13:15:46.831740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.831679    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-openshift-ca\") pod \"d013b999-e4c4-41df-9c4f-961babdabcac\" (UID: \"d013b999-e4c4-41df-9c4f-961babdabcac\") "
Oct 31 13:15:46.831994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.831971    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-openshift-ca" pod "d013b999-e4c4-41df-9c4f-961babdabcac" (UID: "d013b999-e4c4-41df-9c4f-961babdabcac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:46.832172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.832156    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.834366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.834343    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d013b999-e4c4-41df-9c4f-961babdabcac-kube-api-access-fhn6b" pod "d013b999-e4c4-41df-9c4f-961babdabcac" (UID: "d013b999-e4c4-41df-9c4f-961babdabcac"). InnerVolumeSpecName "kube-api-access-fhn6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:46.834366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.834344    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-xks-ca" pod "d013b999-e4c4-41df-9c4f-961babdabcac" (UID: "d013b999-e4c4-41df-9c4f-961babdabcac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:46.933662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.933624    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d013b999-e4c4-41df-9c4f-961babdabcac-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:46.933662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:46.933660    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhn6b\" (UniqueName: \"kubernetes.io/projected/d013b999-e4c4-41df-9c4f-961babdabcac-kube-api-access-fhn6b\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:47.614605 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.614521    5787 generic.go:356] "Generic (PLEG): container finished" podID="bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e" containerID="1abeea7fcaa808964861d2d09b1c465e8a408ebb17d0efa612c1e8b765345f56" exitCode=0
Oct 31 13:15:47.615086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.614656    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf" event={"ID":"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e","Type":"ContainerDied","Data":"1abeea7fcaa808964861d2d09b1c465e8a408ebb17d0efa612c1e8b765345f56"}
Oct 31 13:15:47.620212 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.620190    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"
Oct 31 13:15:47.620335 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.620228    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm" event={"ID":"d013b999-e4c4-41df-9c4f-961babdabcac","Type":"ContainerDied","Data":"077e7d8c577af43b4b2e636893ed7d8a4ef8ba45966fce0756585e478ab6f8d4"}
Oct 31 13:15:47.620335 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.620278    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="077e7d8c577af43b4b2e636893ed7d8a4ef8ba45966fce0756585e478ab6f8d4"
Oct 31 13:15:47.706561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.706526    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main" containerID="cri-o://5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08" gracePeriod=45
Oct 31 13:15:47.839697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.839671    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:47.848032 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.848003    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-xks-ca\") pod \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") "
Oct 31 13:15:47.848212 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.848153    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-openshift-ca\") pod \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") "
Oct 31 13:15:47.848212 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.848203    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a835cb0-8e7f-4ca7-943d-1813763f3a36-kube-api-access-2t7rx\" (UniqueName: \"kubernetes.io/projected/3a835cb0-8e7f-4ca7-943d-1813763f3a36-kube-api-access-2t7rx\") pod \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\" (UID: \"3a835cb0-8e7f-4ca7-943d-1813763f3a36\") "
Oct 31 13:15:47.848506 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.848480    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-openshift-ca" pod "3a835cb0-8e7f-4ca7-943d-1813763f3a36" (UID: "3a835cb0-8e7f-4ca7-943d-1813763f3a36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:47.848777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.848758    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:47.850423 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.850400    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a835cb0-8e7f-4ca7-943d-1813763f3a36-kube-api-access-2t7rx" pod "3a835cb0-8e7f-4ca7-943d-1813763f3a36" (UID: "3a835cb0-8e7f-4ca7-943d-1813763f3a36"). InnerVolumeSpecName "kube-api-access-2t7rx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:47.850506 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.850432    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-xks-ca" pod "3a835cb0-8e7f-4ca7-943d-1813763f3a36" (UID: "3a835cb0-8e7f-4ca7-943d-1813763f3a36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:47.950322 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.950268    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t7rx\" (UniqueName: \"kubernetes.io/projected/3a835cb0-8e7f-4ca7-943d-1813763f3a36-kube-api-access-2t7rx\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:47.950322 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.950300    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a835cb0-8e7f-4ca7-943d-1813763f3a36-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:47.988379 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:47.988352    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:15:48.051129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.051093    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-home\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-home\") pod \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") "
Oct 31 13:15:48.051501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.051271    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-model-cache\") pod \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") "
Oct 31 13:15:48.051501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.051403    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-home" pod "1877b8bc-92f7-41c4-a6d5-88ed2b550001" (UID: "1877b8bc-92f7-41c4-a6d5-88ed2b550001"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:48.051652 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.051537    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-model-cache" pod "1877b8bc-92f7-41c4-a6d5-88ed2b550001" (UID: "1877b8bc-92f7-41c4-a6d5-88ed2b550001"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:48.051652 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.051596    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1877b8bc-92f7-41c4-a6d5-88ed2b550001-kube-api-access-65cqb\" (UniqueName: \"kubernetes.io/projected/1877b8bc-92f7-41c4-a6d5-88ed2b550001-kube-api-access-65cqb\") pod \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") "
Oct 31 13:15:48.051806 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.051692    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tmp-dir\") pod \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") "
Oct 31 13:15:48.051862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.051831    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tls-certs\" (UniqueName: \"kubernetes.io/secret/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tls-certs\") pod \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") "
Oct 31 13:15:48.052149 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.052033    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-dshm\") pod \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\" (UID: \"1877b8bc-92f7-41c4-a6d5-88ed2b550001\") "
Oct 31 13:15:48.052659 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.052627    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tmp-dir" pod "1877b8bc-92f7-41c4-a6d5-88ed2b550001" (UID: "1877b8bc-92f7-41c4-a6d5-88ed2b550001"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:48.052825 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.052804    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:48.053020 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.052997    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:48.053108 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.053026    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:48.056194 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.056147    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tls-certs" pod "1877b8bc-92f7-41c4-a6d5-88ed2b550001" (UID: "1877b8bc-92f7-41c4-a6d5-88ed2b550001"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:48.057281 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.057245    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-dshm" pod "1877b8bc-92f7-41c4-a6d5-88ed2b550001" (UID: "1877b8bc-92f7-41c4-a6d5-88ed2b550001"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:15:48.058758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.058073    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1877b8bc-92f7-41c4-a6d5-88ed2b550001-kube-api-access-65cqb" pod "1877b8bc-92f7-41c4-a6d5-88ed2b550001" (UID: "1877b8bc-92f7-41c4-a6d5-88ed2b550001"). InnerVolumeSpecName "kube-api-access-65cqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:48.154163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.154065    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1877b8bc-92f7-41c4-a6d5-88ed2b550001-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:48.154163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.154095    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65cqb\" (UniqueName: \"kubernetes.io/projected/1877b8bc-92f7-41c4-a6d5-88ed2b550001-kube-api-access-65cqb\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:48.154163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.154111    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1877b8bc-92f7-41c4-a6d5-88ed2b550001-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:48.256512 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:48.256443    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:15:48.667224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.667188    5787 generic.go:356] "Generic (PLEG): container finished" podID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerID="5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08" exitCode=0
Oct 31 13:15:48.683980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.669824    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"
Oct 31 13:15:48.684355 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.670234    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" event={"ID":"1877b8bc-92f7-41c4-a6d5-88ed2b550001","Type":"ContainerDied","Data":"5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08"}
Oct 31 13:15:48.684627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.684569    5787 scope.go:122] "RemoveContainer" containerID="5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08"
Oct 31 13:15:48.686323 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.684524    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6" event={"ID":"1877b8bc-92f7-41c4-a6d5-88ed2b550001","Type":"ContainerDied","Data":"5bf4cf4c586a5f9751ee376ae56b9ab7a8ea065e50351d05956d68faceda8d55"}
Oct 31 13:15:48.688672 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.688643    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl" event={"ID":"3a835cb0-8e7f-4ca7-943d-1813763f3a36","Type":"ContainerDied","Data":"167b7acfa4008f05e80a4f8ebb83ff2d3be723f4f4510436c921a8dadd3a02da"}
Oct 31 13:15:48.688894 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.688878    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="167b7acfa4008f05e80a4f8ebb83ff2d3be723f4f4510436c921a8dadd3a02da"
Oct 31 13:15:48.690672 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.690657    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"
Oct 31 13:15:48.720538 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.720509    5787 scope.go:122] "RemoveContainer" containerID="5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08"
Oct 31 13:15:48.721839 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:15:48.721810    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08\": container with ID starting with 5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08 not found: ID does not exist" containerID="5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08"
Oct 31 13:15:48.721959 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.721884    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08"} err="failed to get container status \"5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08\": rpc error: code = NotFound desc = could not find container \"5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08\": container with ID starting with 5a3357cbba9cfc3e7a1a4e02e37b80626a7a3047a9a7942e0eb48a9bfeef9f08 not found: ID does not exist"
Oct 31 13:15:48.724648 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.724614    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a714d74-2bc2-4eb2-b3f4-5471047b8cf5" path="/var/lib/kubelet/pods/1a714d74-2bc2-4eb2-b3f4-5471047b8cf5/volumes"
Oct 31 13:15:48.725323 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.725304    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" path="/var/lib/kubelet/pods/4e11ad3c-692f-4245-a6d8-68b4f945b3d1/volumes"
Oct 31 13:15:48.725797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.725784    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" path="/var/lib/kubelet/pods/e2436534-3062-4a6d-ba08-93e324b1ec85/volumes"
Oct 31 13:15:48.752084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.752047    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"]
Oct 31 13:15:48.757628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.757593    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-79c6ff/premium-w6-79c6ff-kserve-9476bfd5f-lbmh6"]
Oct 31 13:15:48.880006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:48.879977    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:49.012123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.012083    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-kube-api-access-rb8gd\" (UniqueName: \"kubernetes.io/projected/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-kube-api-access-rb8gd\") pod \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") "
Oct 31 13:15:49.012353 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.012195    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-openshift-ca\") pod \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") "
Oct 31 13:15:49.012420 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.012355    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-xks-ca\") pod \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\" (UID: \"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e\") "
Oct 31 13:15:49.012591 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.012561    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-openshift-ca" pod "bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e" (UID: "bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:49.012944 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.012920    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:49.015123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.015044    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-xks-ca" pod "bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e" (UID: "bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:49.015123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.015086    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-kube-api-access-rb8gd" pod "bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e" (UID: "bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e"). InnerVolumeSpecName "kube-api-access-rb8gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:49.120798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.119819    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8gd\" (UniqueName: \"kubernetes.io/projected/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-kube-api-access-rb8gd\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:49.120798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.119862    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:49.557107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.557066    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"]
Oct 31 13:15:49.700174 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.700140    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"
Oct 31 13:15:49.700174 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.700152    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf" event={"ID":"bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e","Type":"ContainerDied","Data":"94f10d279e0f07d30b6dd08d90a79dfe57725181186695e0596c066d280fbaf8"}
Oct 31 13:15:49.700689 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:49.700200    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f10d279e0f07d30b6dd08d90a79dfe57725181186695e0596c066d280fbaf8"
Oct 31 13:15:50.712713 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:50.712671    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" path="/var/lib/kubelet/pods/1877b8bc-92f7-41c4-a6d5-88ed2b550001/volumes"
Oct 31 13:15:51.094994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.094898    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"]
Oct 31 13:15:51.152867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.152824    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"]
Oct 31 13:15:51.333673 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.333616    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"]
Oct 31 13:15:51.335217 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.335171    5787 cpu_manager.go:414] "containerMap: removing container" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main"
Oct 31 13:15:51.335448 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.335432    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main"
Oct 31 13:15:51.335634 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.335622    5787 cpu_manager.go:414] "containerMap: removing container" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy"
Oct 31 13:15:51.335747 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.335734    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy"
Oct 31 13:15:51.336516 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.335860    5787 cpu_manager.go:414] "containerMap: removing container" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy"
Oct 31 13:15:51.336662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.336646    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy"
Oct 31 13:15:51.336775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.336764    5787 cpu_manager.go:414] "containerMap: removing container" podUID="bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e" containerName="revoke-keys"
Oct 31 13:15:51.336913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.336856    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e" containerName="revoke-keys"
Oct 31 13:15:51.337051 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.337040    5787 cpu_manager.go:414] "containerMap: removing container" podUID="d013b999-e4c4-41df-9c4f-961babdabcac" containerName="revoke-keys"
Oct 31 13:15:51.337132 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.337121    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d013b999-e4c4-41df-9c4f-961babdabcac" containerName="revoke-keys"
Oct 31 13:15:51.337241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.337231    5787 cpu_manager.go:414] "containerMap: removing container" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy"
Oct 31 13:15:51.337312 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.337303    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy"
Oct 31 13:15:51.337389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.337380    5787 cpu_manager.go:414] "containerMap: removing container" podUID="3a835cb0-8e7f-4ca7-943d-1813763f3a36" containerName="revoke-keys"
Oct 31 13:15:51.337460 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.337450    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a835cb0-8e7f-4ca7-943d-1813763f3a36" containerName="revoke-keys"
Oct 31 13:15:51.337747 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.337689    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="321518c6-acce-489e-9cf5-67a29bafc5ad" containerName="istio-proxy"
Oct 31 13:15:51.337870 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.337857    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a835cb0-8e7f-4ca7-943d-1813763f3a36" containerName="revoke-keys"
Oct 31 13:15:51.337968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.337958    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="1877b8bc-92f7-41c4-a6d5-88ed2b550001" containerName="main"
Oct 31 13:15:51.338060 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.338051    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2436534-3062-4a6d-ba08-93e324b1ec85" containerName="istio-proxy"
Oct 31 13:15:51.338145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.338134    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e11ad3c-692f-4245-a6d8-68b4f945b3d1" containerName="istio-proxy"
Oct 31 13:15:51.338221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.338212    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e" containerName="revoke-keys"
Oct 31 13:15:51.338358 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.338349    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="d013b999-e4c4-41df-9c4f-961babdabcac" containerName="revoke-keys"
Oct 31 13:15:51.419308 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.419220    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.425751 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.425438    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:15:51.425751 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.425525    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:15:51.433329 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.433304    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"]
Oct 31 13:15:51.440964 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.440939    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.465448 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.465423    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"]
Oct 31 13:15:51.479036 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.479003    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"]
Oct 31 13:15:51.489083 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.489052    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:15:51.489393 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.489372    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:15:51.515265 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.515214    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"]
Oct 31 13:15:51.543730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.532095    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0add37c5-br6lt\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.565728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.544228    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bls\" (UniqueName: \"kubernetes.io/projected/84e1cdfd-3a07-40bc-958d-874818c2e834-kube-api-access-p9bls\") pod \"maas-api-revoke-sub-0add37c5-br6lt\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.566372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.559199    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.572965 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.572879    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"]
Oct 31 13:15:51.575033 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.574990    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0add37c5-br6lt\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.696107 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.696051    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0add37c5-br6lt\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.697611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.697516    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20d70537-9klbf\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.697855 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.697836    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0add37c5-br6lt\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.697980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.697966    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9400ff4e-p4tgr\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.698179 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.698165    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9400ff4e-p4tgr\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.698301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.698288    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzgbs\" (UniqueName: \"kubernetes.io/projected/0aaeb079-0a80-40c4-bb62-a9767bc80376-kube-api-access-fzgbs\") pod \"maas-api-revoke-sub-9400ff4e-p4tgr\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.698431 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.698415    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bls\" (UniqueName: \"kubernetes.io/projected/84e1cdfd-3a07-40bc-958d-874818c2e834-kube-api-access-p9bls\") pod \"maas-api-revoke-sub-0add37c5-br6lt\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.698603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.698586    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bvm2\" (UniqueName: \"kubernetes.io/projected/c3a859a8-afaa-458f-831b-7e549ce27426-kube-api-access-4bvm2\") pod \"maas-api-revoke-sub-20d70537-9klbf\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.698810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.698794    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20d70537-9klbf\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.699988 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.699964    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0add37c5-br6lt\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.763413 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.763382    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0add37c5-br6lt\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.769844 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.769761    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bls\" (UniqueName: \"kubernetes.io/projected/84e1cdfd-3a07-40bc-958d-874818c2e834-kube-api-access-p9bls\") pod \"maas-api-revoke-sub-0add37c5-br6lt\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:51.801124 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.801084    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20d70537-9klbf\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.801404 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.801383    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9400ff4e-p4tgr\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.801671 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.801646    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9400ff4e-p4tgr\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.801798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.801744    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzgbs\" (UniqueName: \"kubernetes.io/projected/0aaeb079-0a80-40c4-bb62-a9767bc80376-kube-api-access-fzgbs\") pod \"maas-api-revoke-sub-9400ff4e-p4tgr\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.801867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.801849    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bvm2\" (UniqueName: \"kubernetes.io/projected/c3a859a8-afaa-458f-831b-7e549ce27426-kube-api-access-4bvm2\") pod \"maas-api-revoke-sub-20d70537-9klbf\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.801936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.801920    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20d70537-9klbf\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.803049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.803024    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9400ff4e-p4tgr\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.804856 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.804815    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20d70537-9klbf\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.805304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.805282    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20d70537-9klbf\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.805442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.805425    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9400ff4e-p4tgr\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.820565 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.820526    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzgbs\" (UniqueName: \"kubernetes.io/projected/0aaeb079-0a80-40c4-bb62-a9767bc80376-kube-api-access-fzgbs\") pod \"maas-api-revoke-sub-9400ff4e-p4tgr\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:51.840244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.840213    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bvm2\" (UniqueName: \"kubernetes.io/projected/c3a859a8-afaa-458f-831b-7e549ce27426-kube-api-access-4bvm2\") pod \"maas-api-revoke-sub-20d70537-9klbf\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:51.957346 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:51.957265    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:52.052225 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.052123    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:52.068011 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.067784    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:52.293780 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:52.293751    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a859a8_afaa_458f_831b_7e549ce27426.slice/crio-331e5eb741856ea51f90d9673d8007735cebf3132c06fb928f70a4d45bedc962 WatchSource:0}: Error finding container 331e5eb741856ea51f90d9673d8007735cebf3132c06fb928f70a4d45bedc962: Status 404 returned error can't find the container with id 331e5eb741856ea51f90d9673d8007735cebf3132c06fb928f70a4d45bedc962
Oct 31 13:15:52.295870 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.295850    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"]
Oct 31 13:15:52.301995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.301976    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"]
Oct 31 13:15:52.302074 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.302064    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.323139 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.323104    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"]
Oct 31 13:15:52.411986 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.411952    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.411986 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.411986    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.412199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.412114    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.412199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.412150    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.412271 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.412220    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.412271 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.412237    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.412338 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.412302    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.412411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.412393    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.412461 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.412447    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpjq2\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-kube-api-access-tpjq2\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.422363 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:52.422336    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aaeb079_0a80_40c4_bb62_a9767bc80376.slice/crio-f8242807d747a291b92fe782de3ccceb8f6d09fbe4446a89bc8ce9656ecf372f WatchSource:0}: Error finding container f8242807d747a291b92fe782de3ccceb8f6d09fbe4446a89bc8ce9656ecf372f: Status 404 returned error can't find the container with id f8242807d747a291b92fe782de3ccceb8f6d09fbe4446a89bc8ce9656ecf372f
Oct 31 13:15:52.423504 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:52.423479    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e1cdfd_3a07_40bc_958d_874818c2e834.slice/crio-7bffe2a7033c208271ca4c730b77a4d73111c4ddf4c513315211b41e9b27f520 WatchSource:0}: Error finding container 7bffe2a7033c208271ca4c730b77a4d73111c4ddf4c513315211b41e9b27f520: Status 404 returned error can't find the container with id 7bffe2a7033c208271ca4c730b77a4d73111c4ddf4c513315211b41e9b27f520
Oct 31 13:15:52.424675 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.424655    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"]
Oct 31 13:15:52.430267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.430248    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"]
Oct 31 13:15:52.513995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.513956    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpjq2\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-kube-api-access-tpjq2\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514093    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514119    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514194    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514232    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514402 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514287    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514402 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514325    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514418    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514478    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514815 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514767    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-certs\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514856    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-credential-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.514913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514879    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-data\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.515000 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.514930    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-socket\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.515086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.515067    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istiod-ca-cert\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.516770 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.516751    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-envoy\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.517210 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.517194    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-podinfo\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.528184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.528152    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-token\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.528314 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.528246    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpjq2\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-kube-api-access-tpjq2\") pod \"e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") " pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.623122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.623088    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:52.775653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.775617    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr" event={"ID":"0aaeb079-0a80-40c4-bb62-a9767bc80376","Type":"ContainerStarted","Data":"f8242807d747a291b92fe782de3ccceb8f6d09fbe4446a89bc8ce9656ecf372f"}
Oct 31 13:15:52.777124 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.777097    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt" event={"ID":"84e1cdfd-3a07-40bc-958d-874818c2e834","Type":"ContainerStarted","Data":"7bffe2a7033c208271ca4c730b77a4d73111c4ddf4c513315211b41e9b27f520"}
Oct 31 13:15:52.778976 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.778947    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf" event={"ID":"c3a859a8-afaa-458f-831b-7e549ce27426","Type":"ContainerStarted","Data":"3ba7d8cef9905de05d45f737763c34ce1d2aed1fda6d289386608fcf607e7cd2"}
Oct 31 13:15:52.779094 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.779004    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf" event={"ID":"c3a859a8-afaa-458f-831b-7e549ce27426","Type":"ContainerStarted","Data":"331e5eb741856ea51f90d9673d8007735cebf3132c06fb928f70a4d45bedc962"}
Oct 31 13:15:52.804024 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.803912    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf" podStartSLOduration=1.8038946230000001 podStartE2EDuration="1.803894623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:52.799065623 +0000 UTC m=+932.215007465" watchObservedRunningTime="2026-10-31 13:15:52.803894623 +0000 UTC m=+932.219836501"
Oct 31 13:15:52.915232 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:15:52.915189    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56bc1f6_53e8_4962_b690_46ed63a7ee4d.slice/crio-564c4c8f6cbaaa60c7190c16f0834d04eaf5868185a21905b9ea1033b9c67791 WatchSource:0}: Error finding container 564c4c8f6cbaaa60c7190c16f0834d04eaf5868185a21905b9ea1033b9c67791: Status 404 returned error can't find the container with id 564c4c8f6cbaaa60c7190c16f0834d04eaf5868185a21905b9ea1033b9c67791
Oct 31 13:15:52.916020 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.915995    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"]
Oct 31 13:15:52.919677 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.918551    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:15:52.919677 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.918664    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:15:52.919677 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:52.918729    5787 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 13:15:53.790767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:53.790726    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh" event={"ID":"b56bc1f6-53e8-4962-b690-46ed63a7ee4d","Type":"ContainerStarted","Data":"9736247f967dd34bc4e4690d2dd4663bd7adf34a4eed14d5a2732c0288330e29"}
Oct 31 13:15:53.791306 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:53.790793    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh" event={"ID":"b56bc1f6-53e8-4962-b690-46ed63a7ee4d","Type":"ContainerStarted","Data":"564c4c8f6cbaaa60c7190c16f0834d04eaf5868185a21905b9ea1033b9c67791"}
Oct 31 13:15:53.792615 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:53.792589    5787 generic.go:356] "Generic (PLEG): container finished" podID="0aaeb079-0a80-40c4-bb62-a9767bc80376" containerID="b4a35a8fe39d7dd5e5ffdc69421224c09005891f8c68c80fd4672bb88f1c2b30" exitCode=0
Oct 31 13:15:53.792778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:53.792657    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr" event={"ID":"0aaeb079-0a80-40c4-bb62-a9767bc80376","Type":"ContainerDied","Data":"b4a35a8fe39d7dd5e5ffdc69421224c09005891f8c68c80fd4672bb88f1c2b30"}
Oct 31 13:15:53.794390 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:53.794366    5787 generic.go:356] "Generic (PLEG): container finished" podID="84e1cdfd-3a07-40bc-958d-874818c2e834" containerID="71ec49ffae3f6077521d7f3552cb36617b3e01ad27a3de6fee1c137271845f1f" exitCode=0
Oct 31 13:15:53.794465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:53.794440    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt" event={"ID":"84e1cdfd-3a07-40bc-958d-874818c2e834","Type":"ContainerDied","Data":"71ec49ffae3f6077521d7f3552cb36617b3e01ad27a3de6fee1c137271845f1f"}
Oct 31 13:15:53.796223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:53.796202    5787 generic.go:356] "Generic (PLEG): container finished" podID="c3a859a8-afaa-458f-831b-7e549ce27426" containerID="3ba7d8cef9905de05d45f737763c34ce1d2aed1fda6d289386608fcf607e7cd2" exitCode=0
Oct 31 13:15:53.796320 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:53.796267    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf" event={"ID":"c3a859a8-afaa-458f-831b-7e549ce27426","Type":"ContainerDied","Data":"3ba7d8cef9905de05d45f737763c34ce1d2aed1fda6d289386608fcf607e7cd2"}
Oct 31 13:15:53.891124 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:53.890196    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh" podStartSLOduration=1.890171657 podStartE2EDuration="1.890171657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:15:53.82012106 +0000 UTC m=+933.236062901" watchObservedRunningTime="2026-10-31 13:15:53.890171657 +0000 UTC m=+933.306113498"
Oct 31 13:15:54.388307 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:54.388235    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"]
Oct 31 13:15:54.388849 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:54.388805    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy" containerID="cri-o://b740bbdcd7a0954163d955da68a69783d18600fef1a192a6b97535cf2924a51b" gracePeriod=30
Oct 31 13:15:54.416897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:54.416847    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"]
Oct 31 13:15:54.633516 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:54.631131    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:15:54.688809 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:54.688673    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 13:15:54.719856 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:54.717980    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh" podUID="b56bc1f6-53e8-4962-b690-46ed63a7ee4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 13:15:54.850735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:54.849392    5787 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh" secret="" err="secret \"e2e-worker-w1-a2c006-openshift-default-dockercfg-shg42\" not found"
Oct 31 13:15:55.643731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:55.634104    5787 patch_prober.go:29] interesting pod/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Oct 31 13:15:55.643731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:55.634168    5787 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh" podUID="b56bc1f6-53e8-4962-b690-46ed63a7ee4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Oct 31 13:15:56.138958 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:56.115100    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:56.218733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:56.197168    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:56.219268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:56.219044    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x" podUID="a6893107-8ce0-40a6-846e-a7eecfdbe6f5" containerName="payload-processing" containerID="cri-o://e6eb4fb8f1b406c7f5f605af2bf7474635fb42f63b1f2316f4cfae7a4e6b7d90" gracePeriod=25
Oct 31 13:15:56.519760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:56.356230    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:56.519760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:56.357093    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": context deadline exceeded"
Oct 31 13:15:56.519760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:56.357630    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786" podUID="b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" containerName="payload-pre-processing" containerID="cri-o://7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236" gracePeriod=25
Oct 31 13:15:56.519760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:56.455808    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh" podUID="b56bc1f6-53e8-4962-b690-46ed63a7ee4d" containerName="istio-proxy" containerID="cri-o://9736247f967dd34bc4e4690d2dd4663bd7adf34a4eed14d5a2732c0288330e29" gracePeriod=30
Oct 31 13:15:56.519760 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:56.457360    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:56.926673 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:56.457218    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:57.138386 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.137935    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 31 13:15:57.325784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.308524    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:15:57.405090 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.402189    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": context deadline exceeded"
Oct 31 13:15:57.511731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.485835    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:57.544190 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.502292    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:15:57.552991 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.552961    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:15:57.594360 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.594275    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:15:57.594445 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.594372    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:57.793902 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.793847    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:57.801120 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.801083    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:15:57.888119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.888019    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:15:57.895788 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.895747    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:57.899939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:57.897207    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:15:58.024880 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.020993    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:58.115757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.093991    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:58.138990 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.138395    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:58.300946 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.274149    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:15:58.469646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.467520    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:15:58.469646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.468398    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-xks-ca\") pod \"84e1cdfd-3a07-40bc-958d-874818c2e834\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") "
Oct 31 13:15:58.469646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.468480    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aaeb079-0a80-40c4-bb62-a9767bc80376-kube-api-access-fzgbs\" (UniqueName: \"kubernetes.io/projected/0aaeb079-0a80-40c4-bb62-a9767bc80376-kube-api-access-fzgbs\") pod \"0aaeb079-0a80-40c4-bb62-a9767bc80376\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") "
Oct 31 13:15:58.469646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.468617    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-xks-ca\") pod \"0aaeb079-0a80-40c4-bb62-a9767bc80376\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") "
Oct 31 13:15:58.469646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.468666    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-openshift-ca\") pod \"0aaeb079-0a80-40c4-bb62-a9767bc80376\" (UID: \"0aaeb079-0a80-40c4-bb62-a9767bc80376\") "
Oct 31 13:15:58.469646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.468761    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e1cdfd-3a07-40bc-958d-874818c2e834-kube-api-access-p9bls\" (UniqueName: \"kubernetes.io/projected/84e1cdfd-3a07-40bc-958d-874818c2e834-kube-api-access-p9bls\") pod \"84e1cdfd-3a07-40bc-958d-874818c2e834\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") "
Oct 31 13:15:58.469646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.468803    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-openshift-ca\") pod \"84e1cdfd-3a07-40bc-958d-874818c2e834\" (UID: \"84e1cdfd-3a07-40bc-958d-874818c2e834\") "
Oct 31 13:15:58.497844 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.496011    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-openshift-ca" pod "84e1cdfd-3a07-40bc-958d-874818c2e834" (UID: "84e1cdfd-3a07-40bc-958d-874818c2e834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:58.571209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.571162    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-xks-ca\") pod \"c3a859a8-afaa-458f-831b-7e549ce27426\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") "
Oct 31 13:15:58.598782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.597933    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-openshift-ca\") pod \"c3a859a8-afaa-458f-831b-7e549ce27426\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") "
Oct 31 13:15:58.598782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.598162    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a859a8-afaa-458f-831b-7e549ce27426-kube-api-access-4bvm2\" (UniqueName: \"kubernetes.io/projected/c3a859a8-afaa-458f-831b-7e549ce27426-kube-api-access-4bvm2\") pod \"c3a859a8-afaa-458f-831b-7e549ce27426\" (UID: \"c3a859a8-afaa-458f-831b-7e549ce27426\") "
Oct 31 13:15:58.599066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.598942    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:58.602753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.601921    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-openshift-ca" pod "0aaeb079-0a80-40c4-bb62-a9767bc80376" (UID: "0aaeb079-0a80-40c4-bb62-a9767bc80376"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:58.603505 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.603466    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-openshift-ca" pod "c3a859a8-afaa-458f-831b-7e549ce27426" (UID: "c3a859a8-afaa-458f-831b-7e549ce27426"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:15:58.763891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.700392    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:15:58.763891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.701986    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:58.763891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.702013    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:58.763891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.750922    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-xks-ca" pod "c3a859a8-afaa-458f-831b-7e549ce27426" (UID: "c3a859a8-afaa-458f-831b-7e549ce27426"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:58.767665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.767613    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aaeb079-0a80-40c4-bb62-a9767bc80376-kube-api-access-fzgbs" pod "0aaeb079-0a80-40c4-bb62-a9767bc80376" (UID: "0aaeb079-0a80-40c4-bb62-a9767bc80376"). InnerVolumeSpecName "kube-api-access-fzgbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:58.768047 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.768005    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-xks-ca" pod "0aaeb079-0a80-40c4-bb62-a9767bc80376" (UID: "0aaeb079-0a80-40c4-bb62-a9767bc80376"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:58.813783 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.795921    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"
Oct 31 13:15:58.816048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.816007    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzgbs\" (UniqueName: \"kubernetes.io/projected/0aaeb079-0a80-40c4-bb62-a9767bc80376-kube-api-access-fzgbs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:58.816339 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.816320    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aaeb079-0a80-40c4-bb62-a9767bc80376-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:58.816479 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.816464    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a859a8-afaa-458f-831b-7e549ce27426-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:58.827996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.827958    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a859a8-afaa-458f-831b-7e549ce27426-kube-api-access-4bvm2" pod "c3a859a8-afaa-458f-831b-7e549ce27426" (UID: "c3a859a8-afaa-458f-831b-7e549ce27426"). InnerVolumeSpecName "kube-api-access-4bvm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:58.863330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.861791    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-xks-ca" pod "84e1cdfd-3a07-40bc-958d-874818c2e834" (UID: "84e1cdfd-3a07-40bc-958d-874818c2e834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:15:58.863330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.861921    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e1cdfd-3a07-40bc-958d-874818c2e834-kube-api-access-p9bls" pod "84e1cdfd-3a07-40bc-958d-874818c2e834" (UID: "84e1cdfd-3a07-40bc-958d-874818c2e834"). InnerVolumeSpecName "kube-api-access-p9bls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:15:58.894778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.877128    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"
Oct 31 13:15:58.987401 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.951393    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bvm2\" (UniqueName: \"kubernetes.io/projected/c3a859a8-afaa-458f-831b-7e549ce27426-kube-api-access-4bvm2\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:58.987401 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.951424    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9bls\" (UniqueName: \"kubernetes.io/projected/84e1cdfd-3a07-40bc-958d-874818c2e834-kube-api-access-p9bls\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:58.987401 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:58.951440    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84e1cdfd-3a07-40bc-958d-874818c2e834-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:15:59.202529 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.202483    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr" event={"ID":"0aaeb079-0a80-40c4-bb62-a9767bc80376","Type":"ContainerDied","Data":"f8242807d747a291b92fe782de3ccceb8f6d09fbe4446a89bc8ce9656ecf372f"}
Oct 31 13:15:59.230679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.230643    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8242807d747a291b92fe782de3ccceb8f6d09fbe4446a89bc8ce9656ecf372f"
Oct 31 13:15:59.230973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.230954    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:15:59.231135 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.231112    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt" event={"ID":"84e1cdfd-3a07-40bc-958d-874818c2e834","Type":"ContainerDied","Data":"7bffe2a7033c208271ca4c730b77a4d73111c4ddf4c513315211b41e9b27f520"}
Oct 31 13:15:59.231268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.231252    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bffe2a7033c208271ca4c730b77a4d73111c4ddf4c513315211b41e9b27f520"
Oct 31 13:15:59.231399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.231386    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:15:59.231535 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.231523    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:15:59.277830 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.268406    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": context deadline exceeded"
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.645808    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"]
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.652091    5787 cpu_manager.go:414] "containerMap: removing container" podUID="0aaeb079-0a80-40c4-bb62-a9767bc80376" containerName="revoke-keys"
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.652118    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aaeb079-0a80-40c4-bb62-a9767bc80376" containerName="revoke-keys"
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.652152    5787 cpu_manager.go:414] "containerMap: removing container" podUID="c3a859a8-afaa-458f-831b-7e549ce27426" containerName="revoke-keys"
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.652160    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a859a8-afaa-458f-831b-7e549ce27426" containerName="revoke-keys"
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.652208    5787 cpu_manager.go:414] "containerMap: removing container" podUID="84e1cdfd-3a07-40bc-958d-874818c2e834" containerName="revoke-keys"
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.652215    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e1cdfd-3a07-40bc-958d-874818c2e834" containerName="revoke-keys"
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.652377    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a859a8-afaa-458f-831b-7e549ce27426" containerName="revoke-keys"
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.652398    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e1cdfd-3a07-40bc-958d-874818c2e834" containerName="revoke-keys"
Oct 31 13:15:59.654562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.652412    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aaeb079-0a80-40c4-bb62-a9767bc80376" containerName="revoke-keys"
Oct 31 13:15:59.709731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.692527    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"]
Oct 31 13:15:59.709731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.698264    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:15:59.709731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.698928    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:15:59.838632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.837776    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db963afe-4c5pd\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:15:59.838632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.837899    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db963afe-4c5pd\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:15:59.838632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.838036    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf668b65-j2xt6\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:15:59.838632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.838115    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x2v6\" (UniqueName: \"kubernetes.io/projected/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-kube-api-access-8x2v6\") pod \"maas-api-revoke-sub-db963afe-4c5pd\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:15:59.838632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.838166    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf668b65-j2xt6\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:15:59.838632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.838360    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfsfs\" (UniqueName: \"kubernetes.io/projected/cfa8967d-a072-4b5a-8322-0769b0df3578-kube-api-access-jfsfs\") pod \"maas-api-revoke-sub-cf668b65-j2xt6\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:00.011739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.996655    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfsfs\" (UniqueName: \"kubernetes.io/projected/cfa8967d-a072-4b5a-8322-0769b0df3578-kube-api-access-jfsfs\") pod \"maas-api-revoke-sub-cf668b65-j2xt6\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:00.011739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.996762    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db963afe-4c5pd\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:16:00.011739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.996849    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db963afe-4c5pd\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:16:00.011739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.996945    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf668b65-j2xt6\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:00.011739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.996985    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x2v6\" (UniqueName: \"kubernetes.io/projected/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-kube-api-access-8x2v6\") pod \"maas-api-revoke-sub-db963afe-4c5pd\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:16:00.011739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:15:59.997011    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf668b65-j2xt6\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:00.011739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.003828    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db963afe-4c5pd\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:16:00.011739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.005313    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf668b65-j2xt6\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:00.066471 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.066177    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"]
Oct 31 13:16:00.123922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.123809    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:16:00.271778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.261901    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x2v6\" (UniqueName: \"kubernetes.io/projected/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-kube-api-access-8x2v6\") pod \"maas-api-revoke-sub-db963afe-4c5pd\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:16:00.271778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.261982    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfsfs\" (UniqueName: \"kubernetes.io/projected/cfa8967d-a072-4b5a-8322-0769b0df3578-kube-api-access-jfsfs\") pod \"maas-api-revoke-sub-cf668b65-j2xt6\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:00.316910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.282962    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:16:00.316910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.283995    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:16:00.316910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.291974    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:16:00.439787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.392397    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf668b65-j2xt6\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:00.439787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.417790    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db963afe-4c5pd\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:16:00.439787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.425509    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"]
Oct 31 13:16:00.503794 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:00.503676    5787 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aaeb079_0a80_40c4_bb62_a9767bc80376.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a859a8_afaa_458f_831b_7e549ce27426.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527f4c39_43e6_4506_b95d_42423d24c688.slice/crio-b740bbdcd7a0954163d955da68a69783d18600fef1a192a6b97535cf2924a51b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527f4c39_43e6_4506_b95d_42423d24c688.slice/crio-conmon-b740bbdcd7a0954163d955da68a69783d18600fef1a192a6b97535cf2924a51b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e1cdfd_3a07_40bc_958d_874818c2e834.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 13:16:00.533732 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:00.523651    5787 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527f4c39_43e6_4506_b95d_42423d24c688.slice/crio-conmon-b740bbdcd7a0954163d955da68a69783d18600fef1a192a6b97535cf2924a51b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527f4c39_43e6_4506_b95d_42423d24c688.slice/crio-b740bbdcd7a0954163d955da68a69783d18600fef1a192a6b97535cf2924a51b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 13:16:00.595367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.595329    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:16:00.646262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.646227    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"]
Oct 31 13:16:00.647037 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.647002    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-79c6ff-c87b6"]
Oct 31 13:16:00.652388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.652353    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" containerID="cri-o://0ad6d5ed06c872565a89df2be308d475f2176f71d27455fdcf1738851aa1b748" gracePeriod=45
Oct 31 13:16:00.662159 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.662092    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" containerID="cri-o://510e014c0873b6c61dade3316cdce23163efae99e590718a3e657d772dc47079" gracePeriod=45
Oct 31 13:16:00.718007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.717972    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:00.781975 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.781898    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" containerID="cri-o://6f238a6bc40f38c95f0560700798de5483e2c9d11b29f26a0afaaad0787841db" gracePeriod=45
Oct 31 13:16:00.816932 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.816223    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" containerID="cri-o://4f001f00f4e004b942737c7ccc3b75935ab11d6bbe1f3f2ff9fc1b69bbfcdb89" gracePeriod=45
Oct 31 13:16:00.866436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.831313    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"
Oct 31 13:16:00.866436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.832272    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" containerID="cri-o://bb1dfe2236ab043b0d2caca484785e4827076bbe7ef29e92ed6fce1b81486f14" gracePeriod=45
Oct 31 13:16:00.866436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.832756    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": EOF"
Oct 31 13:16:00.902433 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.902052    5787 generic.go:356] "Generic (PLEG): container finished" podID="527f4c39-43e6-4506-b95d-42423d24c688" containerID="b740bbdcd7a0954163d955da68a69783d18600fef1a192a6b97535cf2924a51b" exitCode=0
Oct 31 13:16:00.906827 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.906133    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fc34dd9-ba4c-42f9-97d7-127a9958a9a4" path="/var/lib/kubelet/pods/9fc34dd9-ba4c-42f9-97d7-127a9958a9a4/volumes"
Oct 31 13:16:00.907441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.907169    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:16:00.908059 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.907637    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:16:00.908059 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.907670    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf" event={"ID":"c3a859a8-afaa-458f-831b-7e549ce27426","Type":"ContainerDied","Data":"331e5eb741856ea51f90d9673d8007735cebf3132c06fb928f70a4d45bedc962"}
Oct 31 13:16:00.908059 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.907692    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="331e5eb741856ea51f90d9673d8007735cebf3132c06fb928f70a4d45bedc962"
Oct 31 13:16:00.908382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.908324    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" event={"ID":"527f4c39-43e6-4506-b95d-42423d24c688","Type":"ContainerDied","Data":"b740bbdcd7a0954163d955da68a69783d18600fef1a192a6b97535cf2924a51b"}
Oct 31 13:16:00.926636 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:00.922444    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": EOF"
Oct 31 13:16:01.112263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.112154    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": dial tcp 10.134.0.43:8000: connect: connection refused"
Oct 31 13:16:01.171939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.171267    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": dial tcp 10.134.0.37:8000: connect: connection refused"
Oct 31 13:16:01.171939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.171555    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": dial tcp 10.134.0.91:8000: connect: connection refused"
Oct 31 13:16:01.172227 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.172173    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": dial tcp 10.134.0.72:8000: connect: connection refused"
Oct 31 13:16:01.190663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.190207    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": dial tcp 10.134.0.41:8000: connect: connection refused"
Oct 31 13:16:01.190663 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.190326    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:16:01.222485 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.222408    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:16:01.735697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.734736    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"]
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.770577    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-data\") pod \"527f4c39-43e6-4506-b95d-42423d24c688\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") "
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.770783    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527f4c39-43e6-4506-b95d-42423d24c688-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527f4c39-43e6-4506-b95d-42423d24c688-istiod-ca-cert\") pod \"527f4c39-43e6-4506-b95d-42423d24c688\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") "
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.770873    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-socket\") pod \"527f4c39-43e6-4506-b95d-42423d24c688\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") "
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.770954    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-credential-socket\") pod \"527f4c39-43e6-4506-b95d-42423d24c688\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") "
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.771002    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-certs\") pod \"527f4c39-43e6-4506-b95d-42423d24c688\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") "
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.771133    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-istio-token\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-istio-token\") pod \"527f4c39-43e6-4506-b95d-42423d24c688\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") "
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.771162    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/527f4c39-43e6-4506-b95d-42423d24c688-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527f4c39-43e6-4506-b95d-42423d24c688-istio-podinfo\") pod \"527f4c39-43e6-4506-b95d-42423d24c688\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") "
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.771206    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-envoy\") pod \"527f4c39-43e6-4506-b95d-42423d24c688\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") "
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.771350    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-kube-api-access-f6xtb\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-kube-api-access-f6xtb\") pod \"527f4c39-43e6-4506-b95d-42423d24c688\" (UID: \"527f4c39-43e6-4506-b95d-42423d24c688\") "
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.778268    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-data" pod "527f4c39-43e6-4506-b95d-42423d24c688" (UID: "527f4c39-43e6-4506-b95d-42423d24c688"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:01.808730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.780892    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527f4c39-43e6-4506-b95d-42423d24c688-istiod-ca-cert" pod "527f4c39-43e6-4506-b95d-42423d24c688" (UID: "527f4c39-43e6-4506-b95d-42423d24c688"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:01.828318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.818780    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-socket" pod "527f4c39-43e6-4506-b95d-42423d24c688" (UID: "527f4c39-43e6-4506-b95d-42423d24c688"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:01.854741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.836811    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-credential-socket" pod "527f4c39-43e6-4506-b95d-42423d24c688" (UID: "527f4c39-43e6-4506-b95d-42423d24c688"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:01.854741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.836939    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"]
Oct 31 13:16:01.854741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.853209    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-certs" pod "527f4c39-43e6-4506-b95d-42423d24c688" (UID: "527f4c39-43e6-4506-b95d-42423d24c688"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:01.898758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.890113    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:01.898758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.890148    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/527f4c39-43e6-4506-b95d-42423d24c688-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:01.898758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.890168    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:01.898758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.890182    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:01.898758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.890196    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:01.901478 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.900517    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-kube-api-access-f6xtb" pod "527f4c39-43e6-4506-b95d-42423d24c688" (UID: "527f4c39-43e6-4506-b95d-42423d24c688"). InnerVolumeSpecName "kube-api-access-f6xtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:01.901768 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.901731    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/527f4c39-43e6-4506-b95d-42423d24c688-istio-podinfo" pod "527f4c39-43e6-4506-b95d-42423d24c688" (UID: "527f4c39-43e6-4506-b95d-42423d24c688"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:01.910324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.908614    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-envoy" pod "527f4c39-43e6-4506-b95d-42423d24c688" (UID: "527f4c39-43e6-4506-b95d-42423d24c688"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:01.922896 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.922382    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-istio-token" pod "527f4c39-43e6-4506-b95d-42423d24c688" (UID: "527f4c39-43e6-4506-b95d-42423d24c688"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:01.924489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.924469    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2a262a-k9j7h"]
Oct 31 13:16:01.928770 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.928593    5787 generic.go:356] "Generic (PLEG): container finished" podID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerID="bb1dfe2236ab043b0d2caca484785e4827076bbe7ef29e92ed6fce1b81486f14" exitCode=0
Oct 31 13:16:01.928770 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.928695    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" event={"ID":"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0","Type":"ContainerDied","Data":"bb1dfe2236ab043b0d2caca484785e4827076bbe7ef29e92ed6fce1b81486f14"}
Oct 31 13:16:01.931563 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.931333    5787 generic.go:356] "Generic (PLEG): container finished" podID="144afe47-505e-436b-a07d-5e1ea36818a1" containerID="510e014c0873b6c61dade3316cdce23163efae99e590718a3e657d772dc47079" exitCode=0
Oct 31 13:16:01.931563 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.931506    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" event={"ID":"144afe47-505e-436b-a07d-5e1ea36818a1","Type":"ContainerDied","Data":"510e014c0873b6c61dade3316cdce23163efae99e590718a3e657d772dc47079"}
Oct 31 13:16:01.952758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.937947    5787 generic.go:356] "Generic (PLEG): container finished" podID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerID="4f001f00f4e004b942737c7ccc3b75935ab11d6bbe1f3f2ff9fc1b69bbfcdb89" exitCode=0
Oct 31 13:16:01.952758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:01.938062    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" event={"ID":"4f9a093a-89b7-48ae-9123-d5b87aa9dd27","Type":"ContainerDied","Data":"4f001f00f4e004b942737c7ccc3b75935ab11d6bbe1f3f2ff9fc1b69bbfcdb89"}
Oct 31 13:16:02.053468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.053429    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.053994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.053953    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/527f4c39-43e6-4506-b95d-42423d24c688-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.054175 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.054156    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/527f4c39-43e6-4506-b95d-42423d24c688-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.054307 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.054293    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6xtb\" (UniqueName: \"kubernetes.io/projected/527f4c39-43e6-4506-b95d-42423d24c688-kube-api-access-f6xtb\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.064927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.064888    5787 generic.go:356] "Generic (PLEG): container finished" podID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerID="0ad6d5ed06c872565a89df2be308d475f2176f71d27455fdcf1738851aa1b748" exitCode=0
Oct 31 13:16:02.065122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.064999    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" event={"ID":"ffd95bfe-b694-41a9-bf39-ad654cedee39","Type":"ContainerDied","Data":"0ad6d5ed06c872565a89df2be308d475f2176f71d27455fdcf1738851aa1b748"}
Oct 31 13:16:02.068743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.067916    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7" event={"ID":"527f4c39-43e6-4506-b95d-42423d24c688","Type":"ContainerDied","Data":"21d87e7c8391bf2f784ef38c6291af6cbaa4bc6874d18542cf235fd79e65d4f3"}
Oct 31 13:16:02.068743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.067989    5787 scope.go:122] "RemoveContainer" containerID="b740bbdcd7a0954163d955da68a69783d18600fef1a192a6b97535cf2924a51b"
Oct 31 13:16:02.068743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.068146    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"
Oct 31 13:16:02.127743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.110170    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": dial tcp 10.134.0.43:8000: connect: connection refused"
Oct 31 13:16:02.127743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.110285    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:16:02.201781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.197877    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main" probeResult="failure" output="Get \"https://10.134.0.41:8000/ready\": dial tcp 10.134.0.41:8000: connect: connection refused"
Oct 31 13:16:02.210602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.208693    5787 generic.go:356] "Generic (PLEG): container finished" podID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerID="6f238a6bc40f38c95f0560700798de5483e2c9d11b29f26a0afaaad0787841db" exitCode=0
Oct 31 13:16:02.210602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.208826    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" event={"ID":"9c308e8e-d08e-43d5-accf-e9e9baa052b4","Type":"ContainerDied","Data":"6f238a6bc40f38c95f0560700798de5483e2c9d11b29f26a0afaaad0787841db"}
Oct 31 13:16:02.219288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.216247    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6" event={"ID":"cfa8967d-a072-4b5a-8322-0769b0df3578","Type":"ContainerStarted","Data":"d671325858d0c2cb78c3180b49300c32e0a9360fd6d2d72e88ce58fcc4182148"}
Oct 31 13:16:02.221365 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.221046    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main" probeResult="failure" output="Get \"https://10.134.0.37:8000/ready\": dial tcp 10.134.0.37:8000: connect: connection refused"
Oct 31 13:16:02.221365 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.221158    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:16:02.300287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.296618    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"]
Oct 31 13:16:02.304142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.304111    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-wflr7"]
Oct 31 13:16:02.490655 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.488780    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:16:02.525170 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.513558    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:16:02.602777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.602422    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:16:02.605127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.605057    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:16:02.606002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.605974    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tmp-dir\") pod \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") "
Oct 31 13:16:02.606129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606020    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tls-certs\") pod \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") "
Oct 31 13:16:02.606191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606133    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-home\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-home\") pod \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") "
Oct 31 13:16:02.606191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606168    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-tmp-dir\") pod \"144afe47-505e-436b-a07d-5e1ea36818a1\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") "
Oct 31 13:16:02.606309 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606239    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-dshm\") pod \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") "
Oct 31 13:16:02.606309 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606269    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c308e8e-d08e-43d5-accf-e9e9baa052b4-kube-api-access-wzxt8\" (UniqueName: \"kubernetes.io/projected/9c308e8e-d08e-43d5-accf-e9e9baa052b4-kube-api-access-wzxt8\") pod \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") "
Oct 31 13:16:02.606419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606308    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/144afe47-505e-436b-a07d-5e1ea36818a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/144afe47-505e-436b-a07d-5e1ea36818a1-tls-certs\") pod \"144afe47-505e-436b-a07d-5e1ea36818a1\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") "
Oct 31 13:16:02.606419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606399    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-model-cache\") pod \"144afe47-505e-436b-a07d-5e1ea36818a1\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") "
Oct 31 13:16:02.606525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606462    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-home\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-home\") pod \"144afe47-505e-436b-a07d-5e1ea36818a1\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") "
Oct 31 13:16:02.606525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606503    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-model-cache\") pod \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\" (UID: \"9c308e8e-d08e-43d5-accf-e9e9baa052b4\") "
Oct 31 13:16:02.606652 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606546    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144afe47-505e-436b-a07d-5e1ea36818a1-kube-api-access-s5tq8\" (UniqueName: \"kubernetes.io/projected/144afe47-505e-436b-a07d-5e1ea36818a1-kube-api-access-s5tq8\") pod \"144afe47-505e-436b-a07d-5e1ea36818a1\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") "
Oct 31 13:16:02.607225 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.606678    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-dshm\") pod \"144afe47-505e-436b-a07d-5e1ea36818a1\" (UID: \"144afe47-505e-436b-a07d-5e1ea36818a1\") "
Oct 31 13:16:02.607225 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.607060    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-home" pod "9c308e8e-d08e-43d5-accf-e9e9baa052b4" (UID: "9c308e8e-d08e-43d5-accf-e9e9baa052b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.607934 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.607415    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tmp-dir" pod "9c308e8e-d08e-43d5-accf-e9e9baa052b4" (UID: "9c308e8e-d08e-43d5-accf-e9e9baa052b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.607934 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.607888    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.608108 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.608067    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-tmp-dir" pod "144afe47-505e-436b-a07d-5e1ea36818a1" (UID: "144afe47-505e-436b-a07d-5e1ea36818a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.608456 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.608254    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.608456 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.608269    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-model-cache" pod "9c308e8e-d08e-43d5-accf-e9e9baa052b4" (UID: "9c308e8e-d08e-43d5-accf-e9e9baa052b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.608456 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.608280    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-home" pod "144afe47-505e-436b-a07d-5e1ea36818a1" (UID: "144afe47-505e-436b-a07d-5e1ea36818a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.608742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.608633    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-model-cache" pod "144afe47-505e-436b-a07d-5e1ea36818a1" (UID: "144afe47-505e-436b-a07d-5e1ea36818a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.631790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.630196    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tls-certs" pod "9c308e8e-d08e-43d5-accf-e9e9baa052b4" (UID: "9c308e8e-d08e-43d5-accf-e9e9baa052b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:16:02.631790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.630886    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-dshm" pod "144afe47-505e-436b-a07d-5e1ea36818a1" (UID: "144afe47-505e-436b-a07d-5e1ea36818a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.631790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.630910    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c308e8e-d08e-43d5-accf-e9e9baa052b4-kube-api-access-wzxt8" pod "9c308e8e-d08e-43d5-accf-e9e9baa052b4" (UID: "9c308e8e-d08e-43d5-accf-e9e9baa052b4"). InnerVolumeSpecName "kube-api-access-wzxt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:02.631790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.631002    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-dshm" pod "9c308e8e-d08e-43d5-accf-e9e9baa052b4" (UID: "9c308e8e-d08e-43d5-accf-e9e9baa052b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.631790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.631105    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/144afe47-505e-436b-a07d-5e1ea36818a1-tls-certs" pod "144afe47-505e-436b-a07d-5e1ea36818a1" (UID: "144afe47-505e-436b-a07d-5e1ea36818a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:16:02.633170 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.633139    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144afe47-505e-436b-a07d-5e1ea36818a1-kube-api-access-s5tq8" pod "144afe47-505e-436b-a07d-5e1ea36818a1" (UID: "144afe47-505e-436b-a07d-5e1ea36818a1"). InnerVolumeSpecName "kube-api-access-s5tq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:02.682006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.681974    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:16:02.709288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.709262    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-dshm\") pod \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") "
Oct 31 13:16:02.709609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.709587    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tmp-dir\") pod \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") "
Oct 31 13:16:02.709859 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.709836    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tmp-dir" pod "4f9a093a-89b7-48ae-9123-d5b87aa9dd27" (UID: "4f9a093a-89b7-48ae-9123-d5b87aa9dd27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.709944 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.709845    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-home\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-home\") pod \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") "
Oct 31 13:16:02.709944 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.709921    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-model-cache\") pod \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") "
Oct 31 13:16:02.710058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.709961    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-tmp-dir\") pod \"ffd95bfe-b694-41a9-bf39-ad654cedee39\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") "
Oct 31 13:16:02.710058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.710043    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tls-certs\") pod \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") "
Oct 31 13:16:02.710157 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.710090    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd95bfe-b694-41a9-bf39-ad654cedee39-kube-api-access-blvdm\" (UniqueName: \"kubernetes.io/projected/ffd95bfe-b694-41a9-bf39-ad654cedee39-kube-api-access-blvdm\") pod \"ffd95bfe-b694-41a9-bf39-ad654cedee39\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") "
Oct 31 13:16:02.710206 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.710183    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-dshm\") pod \"ffd95bfe-b694-41a9-bf39-ad654cedee39\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") "
Oct 31 13:16:02.710257 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.710235    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-model-cache\") pod \"ffd95bfe-b694-41a9-bf39-ad654cedee39\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") "
Oct 31 13:16:02.710363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.710346    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-home\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-home\") pod \"ffd95bfe-b694-41a9-bf39-ad654cedee39\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") "
Oct 31 13:16:02.710441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.710386    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-kube-api-access-d8ls5\" (UniqueName: \"kubernetes.io/projected/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-kube-api-access-d8ls5\") pod \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\" (UID: \"4f9a093a-89b7-48ae-9123-d5b87aa9dd27\") "
Oct 31 13:16:02.710441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.710435    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd95bfe-b694-41a9-bf39-ad654cedee39-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd95bfe-b694-41a9-bf39-ad654cedee39-tls-certs\") pod \"ffd95bfe-b694-41a9-bf39-ad654cedee39\" (UID: \"ffd95bfe-b694-41a9-bf39-ad654cedee39\") "
Oct 31 13:16:02.711121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711097    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/144afe47-505e-436b-a07d-5e1ea36818a1-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711125    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711139    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711153    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711167    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5tq8\" (UniqueName: \"kubernetes.io/projected/144afe47-505e-436b-a07d-5e1ea36818a1-kube-api-access-s5tq8\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711180    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711194    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c308e8e-d08e-43d5-accf-e9e9baa052b4-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711206    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/144afe47-505e-436b-a07d-5e1ea36818a1-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711219    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711232    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c308e8e-d08e-43d5-accf-e9e9baa052b4-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.711858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711245    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzxt8\" (UniqueName: \"kubernetes.io/projected/9c308e8e-d08e-43d5-accf-e9e9baa052b4-kube-api-access-wzxt8\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.712184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.711861    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-tmp-dir" pod "ffd95bfe-b694-41a9-bf39-ad654cedee39" (UID: "ffd95bfe-b694-41a9-bf39-ad654cedee39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.712184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.712096    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-model-cache" pod "4f9a093a-89b7-48ae-9123-d5b87aa9dd27" (UID: "4f9a093a-89b7-48ae-9123-d5b87aa9dd27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.712293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.712272    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-model-cache" pod "ffd95bfe-b694-41a9-bf39-ad654cedee39" (UID: "ffd95bfe-b694-41a9-bf39-ad654cedee39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.712503 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.712473    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-home" pod "ffd95bfe-b694-41a9-bf39-ad654cedee39" (UID: "ffd95bfe-b694-41a9-bf39-ad654cedee39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.712786 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.712764    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-home" pod "4f9a093a-89b7-48ae-9123-d5b87aa9dd27" (UID: "4f9a093a-89b7-48ae-9123-d5b87aa9dd27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.714320 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.714299    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-dshm" pod "4f9a093a-89b7-48ae-9123-d5b87aa9dd27" (UID: "4f9a093a-89b7-48ae-9123-d5b87aa9dd27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.715247 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.715219    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-kube-api-access-d8ls5" pod "4f9a093a-89b7-48ae-9123-d5b87aa9dd27" (UID: "4f9a093a-89b7-48ae-9123-d5b87aa9dd27"). InnerVolumeSpecName "kube-api-access-d8ls5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:02.715629 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.715603    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-dshm" pod "ffd95bfe-b694-41a9-bf39-ad654cedee39" (UID: "ffd95bfe-b694-41a9-bf39-ad654cedee39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.715916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.715892    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tls-certs" pod "4f9a093a-89b7-48ae-9123-d5b87aa9dd27" (UID: "4f9a093a-89b7-48ae-9123-d5b87aa9dd27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:16:02.715990 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.715948    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd95bfe-b694-41a9-bf39-ad654cedee39-tls-certs" pod "ffd95bfe-b694-41a9-bf39-ad654cedee39" (UID: "ffd95bfe-b694-41a9-bf39-ad654cedee39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:16:02.716776 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.716757    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd95bfe-b694-41a9-bf39-ad654cedee39-kube-api-access-blvdm" pod "ffd95bfe-b694-41a9-bf39-ad654cedee39" (UID: "ffd95bfe-b694-41a9-bf39-ad654cedee39"). InnerVolumeSpecName "kube-api-access-blvdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:02.717302 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.717282    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527f4c39-43e6-4506-b95d-42423d24c688" path="/var/lib/kubelet/pods/527f4c39-43e6-4506-b95d-42423d24c688/volumes"
Oct 31 13:16:02.717742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.717725    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5817ea86-18de-470f-b84e-8ff152d98f61" path="/var/lib/kubelet/pods/5817ea86-18de-470f-b84e-8ff152d98f61/volumes"
Oct 31 13:16:02.777998 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:16:02.777969    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f701b62_f9ae_4a6a_9313_ecbdf3fe52fa.slice/crio-ce629d1ec58cceac22af0124d2ef5b42ac288d6caef70d58ed1ab01ce21503e5 WatchSource:0}: Error finding container ce629d1ec58cceac22af0124d2ef5b42ac288d6caef70d58ed1ab01ce21503e5: Status 404 returned error can't find the container with id ce629d1ec58cceac22af0124d2ef5b42ac288d6caef70d58ed1ab01ce21503e5
Oct 31 13:16:02.784688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.784650    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"]
Oct 31 13:16:02.812370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.812328    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tmp-dir\") pod \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") "
Oct 31 13:16:02.812502 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.812386    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tls-certs\") pod \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") "
Oct 31 13:16:02.812502 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.812452    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-model-cache\") pod \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") "
Oct 31 13:16:02.812628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.812521    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-home\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-home\") pod \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") "
Oct 31 13:16:02.812628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.812586    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-kube-api-access-svd72\" (UniqueName: \"kubernetes.io/projected/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-kube-api-access-svd72\") pod \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") "
Oct 31 13:16:02.812628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.812606    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-dshm\") pod \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\" (UID: \"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0\") "
Oct 31 13:16:02.812628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.812598    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tmp-dir" pod "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" (UID: "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.812847 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.812818    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-home" pod "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" (UID: "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.812998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.812958    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813131 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813116    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8ls5\" (UniqueName: \"kubernetes.io/projected/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-kube-api-access-d8ls5\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813210    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffd95bfe-b694-41a9-bf39-ad654cedee39-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813281 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813273    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813340    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813405 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813398    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813466 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813458    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813520 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813512    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813567 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813560    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f9a093a-89b7-48ae-9123-d5b87aa9dd27-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813610    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blvdm\" (UniqueName: \"kubernetes.io/projected/ffd95bfe-b694-41a9-bf39-ad654cedee39-kube-api-access-blvdm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813661    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813748 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813737    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffd95bfe-b694-41a9-bf39-ad654cedee39-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.813798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.813148    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-model-cache" pod "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" (UID: "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.814604 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.814583    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tls-certs" pod "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" (UID: "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:16:02.814910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.814893    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-kube-api-access-svd72" pod "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" (UID: "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0"). InnerVolumeSpecName "kube-api-access-svd72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:02.815174 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.815155    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-dshm" pod "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" (UID: "23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:02.915504 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.915471    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.915735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.915679    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.915810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.915736    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.915810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.915753    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svd72\" (UniqueName: \"kubernetes.io/projected/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-kube-api-access-svd72\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:02.915810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:02.915767    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:03.155871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.155812    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:16:03.156034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.155956    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:16:03.178290 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.178254    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"]
Oct 31 13:16:03.183245 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.183201    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:16:03.187394 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.187361    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-004f64-rhh7s"]
Oct 31 13:16:03.233761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.233687    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"
Oct 31 13:16:03.235266 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.233772    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk" event={"ID":"144afe47-505e-436b-a07d-5e1ea36818a1","Type":"ContainerDied","Data":"769e0f85c9adac4b22f2538a72747c4e700e733dbef0cac3d9d3cf14a99cc4b4"}
Oct 31 13:16:03.235489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.235476    5787 scope.go:122] "RemoveContainer" containerID="510e014c0873b6c61dade3316cdce23163efae99e590718a3e657d772dc47079"
Oct 31 13:16:03.237441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.237408    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb" event={"ID":"4f9a093a-89b7-48ae-9123-d5b87aa9dd27","Type":"ContainerDied","Data":"d062c45a8f9094d411d28e0ea5f6dc93f4664d6d9ab8585bb1a496dbe520eb27"}
Oct 31 13:16:03.237740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.237720    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"
Oct 31 13:16:03.246289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.242854    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"
Oct 31 13:16:03.246289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.245578    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6" event={"ID":"ffd95bfe-b694-41a9-bf39-ad654cedee39","Type":"ContainerDied","Data":"e9896c90e2442d40ec7ccc091a64e8d1ed67cf2e62e1b1a8bbd7b090d7b953cb"}
Oct 31 13:16:03.251325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.251293    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs" event={"ID":"9c308e8e-d08e-43d5-accf-e9e9baa052b4","Type":"ContainerDied","Data":"ab1f25b62fadb31f6fec7be598dda6060e310fd7e9496f0fa5ea67a7dcf5334c"}
Oct 31 13:16:03.253104 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.252874    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"
Oct 31 13:16:03.257929 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.257895    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd" event={"ID":"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa","Type":"ContainerStarted","Data":"ce629d1ec58cceac22af0124d2ef5b42ac288d6caef70d58ed1ab01ce21503e5"}
Oct 31 13:16:03.263861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.263232    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6" event={"ID":"cfa8967d-a072-4b5a-8322-0769b0df3578","Type":"ContainerStarted","Data":"4e570ab7319db24476faa4096ea332945acb0e887e6ffaba0b71d5a7399ecf5c"}
Oct 31 13:16:03.270210 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.267354    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969" event={"ID":"23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0","Type":"ContainerDied","Data":"19efd715d3ed78d738db8a91cf03ca033a1d8a98c5db7e1c04e06a0f73fafb55"}
Oct 31 13:16:03.270210 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.267462    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"
Oct 31 13:16:03.316037 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.315969    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6" podStartSLOduration=4.315947933 podStartE2EDuration="4.315947933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:03.301079075 +0000 UTC m=+942.717020916" watchObservedRunningTime="2026-10-31 13:16:03.315947933 +0000 UTC m=+942.731889774"
Oct 31 13:16:03.333390 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.328940    5787 scope.go:122] "RemoveContainer" containerID="4f001f00f4e004b942737c7ccc3b75935ab11d6bbe1f3f2ff9fc1b69bbfcdb89"
Oct 31 13:16:03.346921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.346886    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"]
Oct 31 13:16:03.359572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.359169    5787 scope.go:122] "RemoveContainer" containerID="0ad6d5ed06c872565a89df2be308d475f2176f71d27455fdcf1738851aa1b748"
Oct 31 13:16:03.360176 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.360114    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/distinct-2-w3-004f64-kserve-5b948464c7-wv2bs"]
Oct 31 13:16:03.370799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.370765    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"]
Oct 31 13:16:03.476945 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.379365    5787 scope.go:122] "RemoveContainer" containerID="6f238a6bc40f38c95f0560700798de5483e2c9d11b29f26a0afaaad0787841db"
Oct 31 13:16:03.538049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.537998    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-426787/premium-w4-426787-kserve-55c99d9bb6-w8969"]
Oct 31 13:16:03.634915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.634783    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"]
Oct 31 13:16:03.667660 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.667633    5787 scope.go:122] "RemoveContainer" containerID="bb1dfe2236ab043b0d2caca484785e4827076bbe7ef29e92ed6fce1b81486f14"
Oct 31 13:16:03.822654 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.822618    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-004f64/simulator-w3-004f64-kserve-54dd474994-4xzlb"]
Oct 31 13:16:03.836476 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.836439    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"]
Oct 31 13:16:03.844927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.844877    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-426787/simulator-w4-426787-kserve-588f6d7647-wkvg6"]
Oct 31 13:16:03.850386 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.850355    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"]
Oct 31 13:16:03.855762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.855729    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-426787/distinct-2-w4-426787-kserve-849b5994b-fpqgk"]
Oct 31 13:16:03.874171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.860205    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"]
Oct 31 13:16:03.875771 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:03.875737    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-426787-cqz6l"]
Oct 31 13:16:04.283432 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.283395    5787 generic.go:356] "Generic (PLEG): container finished" podID="3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa" containerID="9a5b64cfe7e7b7c981838fde56fed4e3f54f4ed5f2bcede31ab3831a11ce437c" exitCode=0
Oct 31 13:16:04.283623 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.283474    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd" event={"ID":"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa","Type":"ContainerDied","Data":"9a5b64cfe7e7b7c981838fde56fed4e3f54f4ed5f2bcede31ab3831a11ce437c"}
Oct 31 13:16:04.285407 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.285379    5787 generic.go:356] "Generic (PLEG): container finished" podID="cfa8967d-a072-4b5a-8322-0769b0df3578" containerID="4e570ab7319db24476faa4096ea332945acb0e887e6ffaba0b71d5a7399ecf5c" exitCode=0
Oct 31 13:16:04.285697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.285674    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6" event={"ID":"cfa8967d-a072-4b5a-8322-0769b0df3578","Type":"ContainerDied","Data":"4e570ab7319db24476faa4096ea332945acb0e887e6ffaba0b71d5a7399ecf5c"}
Oct 31 13:16:04.622544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.622441    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main" containerID="cri-o://3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55" gracePeriod=45
Oct 31 13:16:04.713630 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.713576    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" path="/var/lib/kubelet/pods/144afe47-505e-436b-a07d-5e1ea36818a1/volumes"
Oct 31 13:16:04.714491 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.714459    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" path="/var/lib/kubelet/pods/23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0/volumes"
Oct 31 13:16:04.717560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.716619    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" path="/var/lib/kubelet/pods/4f9a093a-89b7-48ae-9123-d5b87aa9dd27/volumes"
Oct 31 13:16:04.717560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.717446    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85edd540-45e5-473b-b1a5-aa38d139b413" path="/var/lib/kubelet/pods/85edd540-45e5-473b-b1a5-aa38d139b413/volumes"
Oct 31 13:16:04.722224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.719948    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" path="/var/lib/kubelet/pods/9c308e8e-d08e-43d5-accf-e9e9baa052b4/volumes"
Oct 31 13:16:04.722224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.720664    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab5ad7fa-3594-4a7a-bc19-56642630e893" path="/var/lib/kubelet/pods/ab5ad7fa-3594-4a7a-bc19-56642630e893/volumes"
Oct 31 13:16:04.722933 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:04.722903    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" path="/var/lib/kubelet/pods/ffd95bfe-b694-41a9-bf39-ad654cedee39/volumes"
Oct 31 13:16:05.175728 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.169584    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:16:05.243790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.243749    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/debe71f3-91f0-48ac-b79b-85472ff7defd-kube-api-access-rcswr\" (UniqueName: \"kubernetes.io/projected/debe71f3-91f0-48ac-b79b-85472ff7defd-kube-api-access-rcswr\") pod \"debe71f3-91f0-48ac-b79b-85472ff7defd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") "
Oct 31 13:16:05.244007 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.243854    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-home\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-home\") pod \"debe71f3-91f0-48ac-b79b-85472ff7defd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") "
Oct 31 13:16:05.244082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.244015    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-dshm\") pod \"debe71f3-91f0-48ac-b79b-85472ff7defd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") "
Oct 31 13:16:05.244082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.244072    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/debe71f3-91f0-48ac-b79b-85472ff7defd-tls-certs\" (UniqueName: \"kubernetes.io/secret/debe71f3-91f0-48ac-b79b-85472ff7defd-tls-certs\") pod \"debe71f3-91f0-48ac-b79b-85472ff7defd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") "
Oct 31 13:16:05.244190 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.244111    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-tmp-dir\") pod \"debe71f3-91f0-48ac-b79b-85472ff7defd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") "
Oct 31 13:16:05.244190 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.244155    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-model-cache\") pod \"debe71f3-91f0-48ac-b79b-85472ff7defd\" (UID: \"debe71f3-91f0-48ac-b79b-85472ff7defd\") "
Oct 31 13:16:05.244962 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.244930    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-model-cache" pod "debe71f3-91f0-48ac-b79b-85472ff7defd" (UID: "debe71f3-91f0-48ac-b79b-85472ff7defd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:05.245151 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.245130    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-home" pod "debe71f3-91f0-48ac-b79b-85472ff7defd" (UID: "debe71f3-91f0-48ac-b79b-85472ff7defd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:05.245737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.245649    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-tmp-dir" pod "debe71f3-91f0-48ac-b79b-85472ff7defd" (UID: "debe71f3-91f0-48ac-b79b-85472ff7defd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:05.259231 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.259186    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/debe71f3-91f0-48ac-b79b-85472ff7defd-kube-api-access-rcswr" pod "debe71f3-91f0-48ac-b79b-85472ff7defd" (UID: "debe71f3-91f0-48ac-b79b-85472ff7defd"). InnerVolumeSpecName "kube-api-access-rcswr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:05.262019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.261977    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-dshm" pod "debe71f3-91f0-48ac-b79b-85472ff7defd" (UID: "debe71f3-91f0-48ac-b79b-85472ff7defd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:05.263846 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.263813    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/debe71f3-91f0-48ac-b79b-85472ff7defd-tls-certs" pod "debe71f3-91f0-48ac-b79b-85472ff7defd" (UID: "debe71f3-91f0-48ac-b79b-85472ff7defd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:16:05.304337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.304301    5787 generic.go:356] "Generic (PLEG): container finished" podID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerID="3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55" exitCode=0
Oct 31 13:16:05.304653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.304633    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"
Oct 31 13:16:05.306145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.305800    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" event={"ID":"debe71f3-91f0-48ac-b79b-85472ff7defd","Type":"ContainerDied","Data":"3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55"}
Oct 31 13:16:05.306145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.305861    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd" event={"ID":"debe71f3-91f0-48ac-b79b-85472ff7defd","Type":"ContainerDied","Data":"206b239f623536772d67803aab123a1eb162734b06a60f17c99596522b76c27d"}
Oct 31 13:16:05.306145 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.305884    5787 scope.go:122] "RemoveContainer" containerID="3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55"
Oct 31 13:16:05.323151 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.323019    5787 scope.go:122] "RemoveContainer" containerID="3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55"
Oct 31 13:16:05.323514 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:05.323445    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55\": container with ID starting with 3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55 not found: ID does not exist" containerID="3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55"
Oct 31 13:16:05.323514 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.323484    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55"} err="failed to get container status \"3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55\": rpc error: code = NotFound desc = could not find container \"3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55\": container with ID starting with 3f03d979025f4c9343975132aededa5e7847a0f35609a66688956110c9c10e55 not found: ID does not exist"
Oct 31 13:16:05.343324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.343247    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"]
Oct 31 13:16:05.346603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.345672    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.346603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.345754    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/debe71f3-91f0-48ac-b79b-85472ff7defd-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.346603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.345770    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.346603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.345784    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.346603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.345800    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcswr\" (UniqueName: \"kubernetes.io/projected/debe71f3-91f0-48ac-b79b-85472ff7defd-kube-api-access-rcswr\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.346603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.345815    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/debe71f3-91f0-48ac-b79b-85472ff7defd-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.351757 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.350343    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7bf66c/simulator-w0-7bf66c-kserve-55564fdfb9-m28nd"]
Oct 31 13:16:05.386244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.385637    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786" podUID="b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:16:05.386244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.385884    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x" podUID="a6893107-8ce0-40a6-846e-a7eecfdbe6f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:16:05.494030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.493674    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:05.541318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.541285    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:16:05.549797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.549763    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa8967d-a072-4b5a-8322-0769b0df3578-kube-api-access-jfsfs\" (UniqueName: \"kubernetes.io/projected/cfa8967d-a072-4b5a-8322-0769b0df3578-kube-api-access-jfsfs\") pod \"cfa8967d-a072-4b5a-8322-0769b0df3578\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") "
Oct 31 13:16:05.549970 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.549857    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-xks-ca\") pod \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") "
Oct 31 13:16:05.550055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.549965    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-openshift-ca\") pod \"cfa8967d-a072-4b5a-8322-0769b0df3578\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") "
Oct 31 13:16:05.550055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.550012    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-openshift-ca\") pod \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") "
Oct 31 13:16:05.550151 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.550099    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-xks-ca\") pod \"cfa8967d-a072-4b5a-8322-0769b0df3578\" (UID: \"cfa8967d-a072-4b5a-8322-0769b0df3578\") "
Oct 31 13:16:05.550999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.550786    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-openshift-ca" pod "3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa" (UID: "3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:05.550999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.550954    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-openshift-ca" pod "cfa8967d-a072-4b5a-8322-0769b0df3578" (UID: "cfa8967d-a072-4b5a-8322-0769b0df3578"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:05.552857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.552771    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa8967d-a072-4b5a-8322-0769b0df3578-kube-api-access-jfsfs" pod "cfa8967d-a072-4b5a-8322-0769b0df3578" (UID: "cfa8967d-a072-4b5a-8322-0769b0df3578"). InnerVolumeSpecName "kube-api-access-jfsfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:05.553146 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.553117    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-xks-ca" pod "3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa" (UID: "3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:16:05.554182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.554043    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-xks-ca" pod "cfa8967d-a072-4b5a-8322-0769b0df3578" (UID: "cfa8967d-a072-4b5a-8322-0769b0df3578"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:16:05.651476 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.651445    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-kube-api-access-8x2v6\" (UniqueName: \"kubernetes.io/projected/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-kube-api-access-8x2v6\") pod \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\" (UID: \"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa\") "
Oct 31 13:16:05.651974 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.651956    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfsfs\" (UniqueName: \"kubernetes.io/projected/cfa8967d-a072-4b5a-8322-0769b0df3578-kube-api-access-jfsfs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.652027 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.651978    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.652027 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.651988    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.652027 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.651998    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.652027 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.652007    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfa8967d-a072-4b5a-8322-0769b0df3578-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:05.653830 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.653812    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-kube-api-access-8x2v6" pod "3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa" (UID: "3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa"). InnerVolumeSpecName "kube-api-access-8x2v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:05.753698 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:05.753659    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x2v6\" (UniqueName: \"kubernetes.io/projected/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa-kube-api-access-8x2v6\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:06.318038 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:06.318006    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"
Oct 31 13:16:06.318490 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:06.318004    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd" event={"ID":"3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa","Type":"ContainerDied","Data":"ce629d1ec58cceac22af0124d2ef5b42ac288d6caef70d58ed1ab01ce21503e5"}
Oct 31 13:16:06.318490 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:06.318153    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce629d1ec58cceac22af0124d2ef5b42ac288d6caef70d58ed1ab01ce21503e5"
Oct 31 13:16:06.319737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:06.319680    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"
Oct 31 13:16:06.319858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:06.319739    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6" event={"ID":"cfa8967d-a072-4b5a-8322-0769b0df3578","Type":"ContainerDied","Data":"d671325858d0c2cb78c3180b49300c32e0a9360fd6d2d72e88ce58fcc4182148"}
Oct 31 13:16:06.319858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:06.319782    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d671325858d0c2cb78c3180b49300c32e0a9360fd6d2d72e88ce58fcc4182148"
Oct 31 13:16:06.706226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:06.706197    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" path="/var/lib/kubelet/pods/debe71f3-91f0-48ac-b79b-85472ff7defd/volumes"
Oct 31 13:16:07.623838 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:07.623804    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:16:07.966561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:07.966514    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"]
Oct 31 13:16:07.966864 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:07.966836    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy" containerID="cri-o://09bc8226dd4192f71df94f9456b128de5c8713362abc2a269e8eb53744eebd18" gracePeriod=30
Oct 31 13:16:07.980211 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:07.980175    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"]
Oct 31 13:16:07.997967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:07.997889    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy" containerID="cri-o://4f0b8a6b8a7a12513e86be3f4dc02176003909f93c3f926f38d02f89f8ebc82a" gracePeriod=30
Oct 31 13:16:08.009854 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:08.009824    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"]
Oct 31 13:16:08.010310 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:08.010280    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy" containerID="cri-o://a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83" gracePeriod=30
Oct 31 13:16:09.214766 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:09.214694    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"]
Oct 31 13:16:09.215696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:09.215661    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy" containerID="cri-o://40c7478010510aadaf3083f9dec1e55926866b66cdaffac4949181502e443713" gracePeriod=30
Oct 31 13:16:09.221843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:09.221814    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"]
Oct 31 13:16:09.222037 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:09.222014    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" containerID="cri-o://142c71f627c40f94389a904d16edd3f684c7ee28e9dfa1d2363b05e90d112fa9" gracePeriod=30
Oct 31 13:16:09.228795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:09.228762    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"]
Oct 31 13:16:09.229051 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:09.229019    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy" containerID="cri-o://4dd6ae5374e13bbc98fff885b4642abd98e04590a1b2d2b56c3fae2badc6b98e" gracePeriod=30
Oct 31 13:16:10.421942 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.421905    5787 generic.go:356] "Generic (PLEG): container finished" podID="b56bc1f6-53e8-4962-b690-46ed63a7ee4d" containerID="9736247f967dd34bc4e4690d2dd4663bd7adf34a4eed14d5a2732c0288330e29" exitCode=0
Oct 31 13:16:10.422353 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.421981    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh" event={"ID":"b56bc1f6-53e8-4962-b690-46ed63a7ee4d","Type":"ContainerDied","Data":"9736247f967dd34bc4e4690d2dd4663bd7adf34a4eed14d5a2732c0288330e29"}
Oct 31 13:16:10.584133 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.584107    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:16:10.682397 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682304    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-envoy\") pod \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") "
Oct 31 13:16:10.682583 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682436    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-credential-socket\") pod \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") "
Oct 31 13:16:10.682583 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682545    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-data\") pod \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") "
Oct 31 13:16:10.682744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682603    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-certs\") pod \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") "
Oct 31 13:16:10.682744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682639    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istiod-ca-cert\") pod \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") "
Oct 31 13:16:10.682744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682693    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-podinfo\") pod \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") "
Oct 31 13:16:10.682861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682750    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-token\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-token\") pod \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") "
Oct 31 13:16:10.682861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682780    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-kube-api-access-tpjq2\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-kube-api-access-tpjq2\") pod \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") "
Oct 31 13:16:10.682861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682830    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-socket\") pod \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\" (UID: \"b56bc1f6-53e8-4962-b690-46ed63a7ee4d\") "
Oct 31 13:16:10.683363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682820    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-credential-socket" pod "b56bc1f6-53e8-4962-b690-46ed63a7ee4d" (UID: "b56bc1f6-53e8-4962-b690-46ed63a7ee4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:10.683363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.682937    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-certs" pod "b56bc1f6-53e8-4962-b690-46ed63a7ee4d" (UID: "b56bc1f6-53e8-4962-b690-46ed63a7ee4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:10.683558 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.683261    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-data" pod "b56bc1f6-53e8-4962-b690-46ed63a7ee4d" (UID: "b56bc1f6-53e8-4962-b690-46ed63a7ee4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:10.683558 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.683274    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istiod-ca-cert" pod "b56bc1f6-53e8-4962-b690-46ed63a7ee4d" (UID: "b56bc1f6-53e8-4962-b690-46ed63a7ee4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:10.683558 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.683458    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-socket" pod "b56bc1f6-53e8-4962-b690-46ed63a7ee4d" (UID: "b56bc1f6-53e8-4962-b690-46ed63a7ee4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:10.683668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.683629    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:10.683668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.683651    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:10.683668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.683661    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:10.683797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.683669    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:10.683797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.683677    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:10.684830 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.684801    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-envoy" pod "b56bc1f6-53e8-4962-b690-46ed63a7ee4d" (UID: "b56bc1f6-53e8-4962-b690-46ed63a7ee4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:10.685158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.685141    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-token" pod "b56bc1f6-53e8-4962-b690-46ed63a7ee4d" (UID: "b56bc1f6-53e8-4962-b690-46ed63a7ee4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:10.685834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.685805    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-kube-api-access-tpjq2" pod "b56bc1f6-53e8-4962-b690-46ed63a7ee4d" (UID: "b56bc1f6-53e8-4962-b690-46ed63a7ee4d"). InnerVolumeSpecName "kube-api-access-tpjq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:10.685936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.685920    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-podinfo" pod "b56bc1f6-53e8-4962-b690-46ed63a7ee4d" (UID: "b56bc1f6-53e8-4962-b690-46ed63a7ee4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:10.796785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.796742    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:10.796785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.796785    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:10.796785 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.796802    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpjq2\" (UniqueName: \"kubernetes.io/projected/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-kube-api-access-tpjq2\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:10.797120 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.796821    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56bc1f6-53e8-4962-b690-46ed63a7ee4d-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:10.959410 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.959373    5787 patch_prober.go:29] interesting pod/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:16:10.959584 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:10.959440    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:16:11.432371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:11.432294    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"
Oct 31 13:16:11.432371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:11.432310    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh" event={"ID":"b56bc1f6-53e8-4962-b690-46ed63a7ee4d","Type":"ContainerDied","Data":"564c4c8f6cbaaa60c7190c16f0834d04eaf5868185a21905b9ea1033b9c67791"}
Oct 31 13:16:11.432885 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:11.432381    5787 scope.go:122] "RemoveContainer" containerID="9736247f967dd34bc4e4690d2dd4663bd7adf34a4eed14d5a2732c0288330e29"
Oct 31 13:16:11.466787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:11.466749    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"]
Oct 31 13:16:11.471389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:11.471362    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a2c006-openshift-default-585cfbf5b7-npcqh"]
Oct 31 13:16:12.095729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:12.079068    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"]
Oct 31 13:16:12.095729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:12.093096    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"]
Oct 31 13:16:12.095729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:12.093352    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy" containerID="cri-o://f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9" gracePeriod=30
Oct 31 13:16:12.095729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:12.094197    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy" containerID="cri-o://49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16" gracePeriod=30
Oct 31 13:16:12.101781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:12.101752    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"]
Oct 31 13:16:12.102002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:12.101968    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy" containerID="cri-o://6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e" gracePeriod=30
Oct 31 13:16:12.707581 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:12.707550    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56bc1f6-53e8-4962-b690-46ed63a7ee4d" path="/var/lib/kubelet/pods/b56bc1f6-53e8-4962-b690-46ed63a7ee4d/volumes"
Oct 31 13:16:12.936356 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:12.936318    5787 patch_prober.go:29] interesting pod/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:16:12.936531 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:12.936379    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:16:13.065017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.064935    5787 patch_prober.go:29] interesting pod/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:16:13.065343 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.065301    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:16:13.100406 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.093576    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"]
Oct 31 13:16:13.100406 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.093852    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy" containerID="cri-o://f067cb8e433ecf1b49d85f38799fa2964bed65959a8c234f03beb144a744df1f" gracePeriod=30
Oct 31 13:16:13.103892 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.103839    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"]
Oct 31 13:16:13.114721 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.114637    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy" containerID="cri-o://f1a4b404ccdaca53844fafb7a55b0fbbdfea74b1e136bc2fe97087b9742fd770" gracePeriod=30
Oct 31 13:16:13.128042 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.127687    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"]
Oct 31 13:16:13.128042 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.128010    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy" containerID="cri-o://d8ef1b7043a1afc31c3ac72c40c3aa27744a7ea7276db0a8061c35bee2371ef5" gracePeriod=30
Oct 31 13:16:13.459973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.459563    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:16:13.475399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.475356    5787 generic.go:356] "Generic (PLEG): container finished" podID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerID="4f0b8a6b8a7a12513e86be3f4dc02176003909f93c3f926f38d02f89f8ebc82a" exitCode=0
Oct 31 13:16:13.475796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.475766    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" event={"ID":"365f64c7-a490-4ee0-87e1-56c68d22d5f3","Type":"ContainerDied","Data":"4f0b8a6b8a7a12513e86be3f4dc02176003909f93c3f926f38d02f89f8ebc82a"}
Oct 31 13:16:13.478197 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.478166    5787 generic.go:356] "Generic (PLEG): container finished" podID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerID="a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83" exitCode=0
Oct 31 13:16:13.479253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.479224    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"
Oct 31 13:16:13.479682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.479498    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" event={"ID":"0e8209df-eac1-4874-a784-fe7e9a33a7fa","Type":"ContainerDied","Data":"a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83"}
Oct 31 13:16:13.479891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.479864    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv" event={"ID":"0e8209df-eac1-4874-a784-fe7e9a33a7fa","Type":"ContainerDied","Data":"6f5ab0540ecba6a64e2db307dc809089f1e00cdbbbe902d0ac934ef1c7369c42"}
Oct 31 13:16:13.480014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.479912    5787 scope.go:122] "RemoveContainer" containerID="a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83"
Oct 31 13:16:13.481371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.481329    5787 generic.go:356] "Generic (PLEG): container finished" podID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerID="09bc8226dd4192f71df94f9456b128de5c8713362abc2a269e8eb53744eebd18" exitCode=0
Oct 31 13:16:13.481497 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.481447    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" event={"ID":"cca15df2-3a97-4142-8a38-8a0053421cf7","Type":"ContainerDied","Data":"09bc8226dd4192f71df94f9456b128de5c8713362abc2a269e8eb53744eebd18"}
Oct 31 13:16:13.511653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.511544    5787 scope.go:122] "RemoveContainer" containerID="a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83"
Oct 31 13:16:13.512090 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:13.511947    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83\": container with ID starting with a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83 not found: ID does not exist" containerID="a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83"
Oct 31 13:16:13.512090 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.511986    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83"} err="failed to get container status \"a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83\": rpc error: code = NotFound desc = could not find container \"a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83\": container with ID starting with a553b476c094887b39487fc4c8e937865c14c9278b2beddbf30ec9dceb3e9c83 not found: ID does not exist"
Oct 31 13:16:13.512090 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.512010    5787 scope.go:122] "RemoveContainer" containerID="b0023782a8c0b6b535aa17d10da902c9548ba2bc786c6a4cb138074a85a641d8"
Oct 31 13:16:13.544121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.543967    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:16:13.572540 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.572496    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-credential-socket\") pod \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") "
Oct 31 13:16:13.573192 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.573057    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-credential-socket" pod "0e8209df-eac1-4874-a784-fe7e9a33a7fa" (UID: "0e8209df-eac1-4874-a784-fe7e9a33a7fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.573791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.573772    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-token\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-token\") pod \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") "
Oct 31 13:16:13.574445 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.574427    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-kube-api-access-cqpvk\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-kube-api-access-cqpvk\") pod \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") "
Oct 31 13:16:13.576164 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.576146    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-podinfo\") pod \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") "
Oct 31 13:16:13.576330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.576316    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-token\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-token\") pod \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") "
Oct 31 13:16:13.576765 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.576749    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-socket\") pod \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") "
Oct 31 13:16:13.577393 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.577377    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-envoy\") pod \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") "
Oct 31 13:16:13.577838 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.577822    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-credential-socket\") pod \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") "
Oct 31 13:16:13.578270 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.578254    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-data\") pod \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") "
Oct 31 13:16:13.578796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.578778    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-kube-api-access-6sjj4\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-kube-api-access-6sjj4\") pod \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") "
Oct 31 13:16:13.579264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.577185    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-socket" pod "365f64c7-a490-4ee0-87e1-56c68d22d5f3" (UID: "365f64c7-a490-4ee0-87e1-56c68d22d5f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.579264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.577275    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-kube-api-access-cqpvk" pod "365f64c7-a490-4ee0-87e1-56c68d22d5f3" (UID: "365f64c7-a490-4ee0-87e1-56c68d22d5f3"). InnerVolumeSpecName "kube-api-access-cqpvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:13.579264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.578147    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-credential-socket" pod "365f64c7-a490-4ee0-87e1-56c68d22d5f3" (UID: "365f64c7-a490-4ee0-87e1-56c68d22d5f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.579264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.578669    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-data" pod "0e8209df-eac1-4874-a784-fe7e9a33a7fa" (UID: "0e8209df-eac1-4874-a784-fe7e9a33a7fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.579816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.579799    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-envoy\") pod \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") "
Oct 31 13:16:13.580288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.580272    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-certs\") pod \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") "
Oct 31 13:16:13.581095 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.580793    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-podinfo\") pod \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") "
Oct 31 13:16:13.581289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.580728    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-certs" pod "0e8209df-eac1-4874-a784-fe7e9a33a7fa" (UID: "0e8209df-eac1-4874-a784-fe7e9a33a7fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.581592 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.581576    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istiod-ca-cert\") pod \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") "
Oct 31 13:16:13.582184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.582088    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istiod-ca-cert" pod "0e8209df-eac1-4874-a784-fe7e9a33a7fa" (UID: "0e8209df-eac1-4874-a784-fe7e9a33a7fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:13.582477 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.582460    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-socket\") pod \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\" (UID: \"0e8209df-eac1-4874-a784-fe7e9a33a7fa\") "
Oct 31 13:16:13.582955 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.582859    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-socket" pod "0e8209df-eac1-4874-a784-fe7e9a33a7fa" (UID: "0e8209df-eac1-4874-a784-fe7e9a33a7fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.583127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.583110    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-data\") pod \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") "
Oct 31 13:16:13.583670 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.583653    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istiod-ca-cert\") pod \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") "
Oct 31 13:16:13.584134 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.584117    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-certs\") pod \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\" (UID: \"365f64c7-a490-4ee0-87e1-56c68d22d5f3\") "
Oct 31 13:16:13.585166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.583603    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-data" pod "365f64c7-a490-4ee0-87e1-56c68d22d5f3" (UID: "365f64c7-a490-4ee0-87e1-56c68d22d5f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.585301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.584061    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istiod-ca-cert" pod "365f64c7-a490-4ee0-87e1-56c68d22d5f3" (UID: "365f64c7-a490-4ee0-87e1-56c68d22d5f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:13.585375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.584467    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-certs" pod "365f64c7-a490-4ee0-87e1-56c68d22d5f3" (UID: "365f64c7-a490-4ee0-87e1-56c68d22d5f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.585900 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.585696    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-token" pod "365f64c7-a490-4ee0-87e1-56c68d22d5f3" (UID: "365f64c7-a490-4ee0-87e1-56c68d22d5f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:13.585900 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.585857    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-token" pod "0e8209df-eac1-4874-a784-fe7e9a33a7fa" (UID: "0e8209df-eac1-4874-a784-fe7e9a33a7fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:13.585900 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.585873    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-podinfo" pod "365f64c7-a490-4ee0-87e1-56c68d22d5f3" (UID: "365f64c7-a490-4ee0-87e1-56c68d22d5f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:13.587159 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.587130    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-kube-api-access-6sjj4" pod "0e8209df-eac1-4874-a784-fe7e9a33a7fa" (UID: "0e8209df-eac1-4874-a784-fe7e9a33a7fa"). InnerVolumeSpecName "kube-api-access-6sjj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:13.587548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.587521    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-podinfo" pod "0e8209df-eac1-4874-a784-fe7e9a33a7fa" (UID: "0e8209df-eac1-4874-a784-fe7e9a33a7fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:13.587649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.587634    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.587822 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.587806    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.587936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.587922    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.588084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.587958    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.588084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.587977    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.588084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.587817    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-envoy" pod "0e8209df-eac1-4874-a784-fe7e9a33a7fa" (UID: "0e8209df-eac1-4874-a784-fe7e9a33a7fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.588084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.587992    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.588084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.588006    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.588084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.588020    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqpvk\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-kube-api-access-cqpvk\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.588816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.588621    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-envoy" pod "365f64c7-a490-4ee0-87e1-56c68d22d5f3" (UID: "365f64c7-a490-4ee0-87e1-56c68d22d5f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.689039 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689003    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.689263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689238    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.689263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689263    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.689400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689277    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.689400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689290    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sjj4\" (UniqueName: \"kubernetes.io/projected/0e8209df-eac1-4874-a784-fe7e9a33a7fa-kube-api-access-6sjj4\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.689400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689307    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.689400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689321    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e8209df-eac1-4874-a784-fe7e9a33a7fa-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.689400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689335    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.689400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689349    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/365f64c7-a490-4ee0-87e1-56c68d22d5f3-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.689400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.689362    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/365f64c7-a490-4ee0-87e1-56c68d22d5f3-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.719197 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.719163    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:16:13.790741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.790135    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-socket\") pod \"cca15df2-3a97-4142-8a38-8a0053421cf7\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") "
Oct 31 13:16:13.790741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.790249    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-data\") pod \"cca15df2-3a97-4142-8a38-8a0053421cf7\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") "
Oct 31 13:16:13.790741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.790282    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-token\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-token\") pod \"cca15df2-3a97-4142-8a38-8a0053421cf7\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") "
Oct 31 13:16:13.790741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.790328    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cca15df2-3a97-4142-8a38-8a0053421cf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca15df2-3a97-4142-8a38-8a0053421cf7-istiod-ca-cert\") pod \"cca15df2-3a97-4142-8a38-8a0053421cf7\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") "
Oct 31 13:16:13.790741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.790372    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-envoy\") pod \"cca15df2-3a97-4142-8a38-8a0053421cf7\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") "
Oct 31 13:16:13.790741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.790436    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-podinfo\") pod \"cca15df2-3a97-4142-8a38-8a0053421cf7\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") "
Oct 31 13:16:13.790741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.790471    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-certs\") pod \"cca15df2-3a97-4142-8a38-8a0053421cf7\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") "
Oct 31 13:16:13.790741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.790500    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-kube-api-access-42xv9\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-kube-api-access-42xv9\") pod \"cca15df2-3a97-4142-8a38-8a0053421cf7\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") "
Oct 31 13:16:13.790741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.790553    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-credential-socket\") pod \"cca15df2-3a97-4142-8a38-8a0053421cf7\" (UID: \"cca15df2-3a97-4142-8a38-8a0053421cf7\") "
Oct 31 13:16:13.791262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.791095    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-data" pod "cca15df2-3a97-4142-8a38-8a0053421cf7" (UID: "cca15df2-3a97-4142-8a38-8a0053421cf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.792419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.791374    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-certs" pod "cca15df2-3a97-4142-8a38-8a0053421cf7" (UID: "cca15df2-3a97-4142-8a38-8a0053421cf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.792419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.791426    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-socket" pod "cca15df2-3a97-4142-8a38-8a0053421cf7" (UID: "cca15df2-3a97-4142-8a38-8a0053421cf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.792419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.791651    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-credential-socket" pod "cca15df2-3a97-4142-8a38-8a0053421cf7" (UID: "cca15df2-3a97-4142-8a38-8a0053421cf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.792419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.791941    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cca15df2-3a97-4142-8a38-8a0053421cf7-istiod-ca-cert" pod "cca15df2-3a97-4142-8a38-8a0053421cf7" (UID: "cca15df2-3a97-4142-8a38-8a0053421cf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:13.793184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.793156    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.793324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.793190    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.793324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.793206    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cca15df2-3a97-4142-8a38-8a0053421cf7-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.793324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.793220    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.793324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.793234    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.794667 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.794641    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-envoy" pod "cca15df2-3a97-4142-8a38-8a0053421cf7" (UID: "cca15df2-3a97-4142-8a38-8a0053421cf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:13.794808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.794675    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-kube-api-access-42xv9" pod "cca15df2-3a97-4142-8a38-8a0053421cf7" (UID: "cca15df2-3a97-4142-8a38-8a0053421cf7"). InnerVolumeSpecName "kube-api-access-42xv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:13.794886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.794869    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-token" pod "cca15df2-3a97-4142-8a38-8a0053421cf7" (UID: "cca15df2-3a97-4142-8a38-8a0053421cf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:13.796141 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.796112    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-podinfo" pod "cca15df2-3a97-4142-8a38-8a0053421cf7" (UID: "cca15df2-3a97-4142-8a38-8a0053421cf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:13.820489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.820448    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"]
Oct 31 13:16:13.827163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.827127    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-fk8jv"]
Oct 31 13:16:13.894790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.894756    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.894790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.894790    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.895001 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.894802    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42xv9\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-kube-api-access-42xv9\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:13.895001 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:13.894813    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cca15df2-3a97-4142-8a38-8a0053421cf7-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.517831 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.517790    5787 generic.go:356] "Generic (PLEG): container finished" podID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerID="142c71f627c40f94389a904d16edd3f684c7ee28e9dfa1d2363b05e90d112fa9" exitCode=0
Oct 31 13:16:14.518046 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.518023    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" event={"ID":"9206bfbe-392b-4a0e-972e-618cbafa13f9","Type":"ContainerDied","Data":"142c71f627c40f94389a904d16edd3f684c7ee28e9dfa1d2363b05e90d112fa9"}
Oct 31 13:16:14.520975 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.520940    5787 generic.go:356] "Generic (PLEG): container finished" podID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerID="4dd6ae5374e13bbc98fff885b4642abd98e04590a1b2d2b56c3fae2badc6b98e" exitCode=0
Oct 31 13:16:14.521173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.521011    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" event={"ID":"27fe87e5-d534-4f37-8260-5d5b415df0ff","Type":"ContainerDied","Data":"4dd6ae5374e13bbc98fff885b4642abd98e04590a1b2d2b56c3fae2badc6b98e"}
Oct 31 13:16:14.521422 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.521401    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:16:14.523834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.523813    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"
Oct 31 13:16:14.524463 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.523845    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx" event={"ID":"cca15df2-3a97-4142-8a38-8a0053421cf7","Type":"ContainerDied","Data":"8585f7ed7f2646daee016f2c2280ea756543e2c899217be60049e8124444bc48"}
Oct 31 13:16:14.524463 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.523893    5787 scope.go:122] "RemoveContainer" containerID="09bc8226dd4192f71df94f9456b128de5c8713362abc2a269e8eb53744eebd18"
Oct 31 13:16:14.536180 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.535808    5787 generic.go:356] "Generic (PLEG): container finished" podID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerID="40c7478010510aadaf3083f9dec1e55926866b66cdaffac4949181502e443713" exitCode=0
Oct 31 13:16:14.536180 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.536032    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"
Oct 31 13:16:14.536180 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.536068    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls" event={"ID":"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b","Type":"ContainerDied","Data":"40c7478010510aadaf3083f9dec1e55926866b66cdaffac4949181502e443713"}
Oct 31 13:16:14.538571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.538494    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"
Oct 31 13:16:14.538571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.538515    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl" event={"ID":"365f64c7-a490-4ee0-87e1-56c68d22d5f3","Type":"ContainerDied","Data":"e81a33b864ae6450af6caa4e28ffa7ebab8045c3bc16b5ef4014640b0eafb98d"}
Oct 31 13:16:14.566055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.566018    5787 scope.go:122] "RemoveContainer" containerID="40c7478010510aadaf3083f9dec1e55926866b66cdaffac4949181502e443713"
Oct 31 13:16:14.580470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.580368    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:16:14.586698 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.586662    5787 scope.go:122] "RemoveContainer" containerID="4f0b8a6b8a7a12513e86be3f4dc02176003909f93c3f926f38d02f89f8ebc82a"
Oct 31 13:16:14.594054 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.594027    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"]
Oct 31 13:16:14.599763 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.599674    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-pdqzx"]
Oct 31 13:16:14.600650 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.600633    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:16:14.604904 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.604877    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"]
Oct 31 13:16:14.609099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609077    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-kube-api-access-l2b6j\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-kube-api-access-l2b6j\") pod \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") "
Oct 31 13:16:14.609226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609113    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-envoy\") pod \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") "
Oct 31 13:16:14.609226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609158    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-socket\") pod \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") "
Oct 31 13:16:14.609371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609254    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-79c6ff-openshift-default-6bc6b55775-hvxgl"]
Oct 31 13:16:14.609371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609269    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istiod-ca-cert\") pod \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") "
Oct 31 13:16:14.609476 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609439    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-socket" pod "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" (UID: "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.609542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609522    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-data\") pod \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") "
Oct 31 13:16:14.609644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609631    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-podinfo\") pod \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") "
Oct 31 13:16:14.609725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609664    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-token\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-token\") pod \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") "
Oct 31 13:16:14.609725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609667    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istiod-ca-cert" pod "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" (UID: "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:14.609829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609743    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-credential-socket\") pod \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") "
Oct 31 13:16:14.609879 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609840    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-data" pod "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" (UID: "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.609879 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.609872    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-certs\") pod \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\" (UID: \"4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b\") "
Oct 31 13:16:14.610210 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.610190    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-credential-socket" pod "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" (UID: "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.610332 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.610233    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-certs" pod "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" (UID: "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.610443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.610429    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.610501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.610451    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.610501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.610464    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.610501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.610478    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.610501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.610492    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.612158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.612137    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-envoy" pod "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" (UID: "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.612371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.612356    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-podinfo" pod "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" (UID: "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:14.612594 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.612580    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-token" pod "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" (UID: "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:14.613467 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.613453    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-kube-api-access-l2b6j" pod "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" (UID: "4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b"). InnerVolumeSpecName "kube-api-access-l2b6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:14.706412 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.706380    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" path="/var/lib/kubelet/pods/0e8209df-eac1-4874-a784-fe7e9a33a7fa/volumes"
Oct 31 13:16:14.706885 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.706872    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" path="/var/lib/kubelet/pods/365f64c7-a490-4ee0-87e1-56c68d22d5f3/volumes"
Oct 31 13:16:14.707287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.707274    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" path="/var/lib/kubelet/pods/cca15df2-3a97-4142-8a38-8a0053421cf7/volumes"
Oct 31 13:16:14.711036 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711013    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-podinfo\") pod \"27fe87e5-d534-4f37-8260-5d5b415df0ff\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") "
Oct 31 13:16:14.711166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711065    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9206bfbe-392b-4a0e-972e-618cbafa13f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9206bfbe-392b-4a0e-972e-618cbafa13f9-istiod-ca-cert\") pod \"9206bfbe-392b-4a0e-972e-618cbafa13f9\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") "
Oct 31 13:16:14.711166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711102    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-envoy\") pod \"27fe87e5-d534-4f37-8260-5d5b415df0ff\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") "
Oct 31 13:16:14.711166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711121    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-token\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-token\") pod \"27fe87e5-d534-4f37-8260-5d5b415df0ff\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") "
Oct 31 13:16:14.711166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711154    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-certs\") pod \"9206bfbe-392b-4a0e-972e-618cbafa13f9\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") "
Oct 31 13:16:14.711372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711195    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27fe87e5-d534-4f37-8260-5d5b415df0ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fe87e5-d534-4f37-8260-5d5b415df0ff-istiod-ca-cert\") pod \"27fe87e5-d534-4f37-8260-5d5b415df0ff\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") "
Oct 31 13:16:14.711372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711273    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-kube-api-access-qc7km\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-kube-api-access-qc7km\") pod \"9206bfbe-392b-4a0e-972e-618cbafa13f9\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") "
Oct 31 13:16:14.711372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711321    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-data\") pod \"27fe87e5-d534-4f37-8260-5d5b415df0ff\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") "
Oct 31 13:16:14.711372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711356    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-kube-api-access-v5m94\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-kube-api-access-v5m94\") pod \"27fe87e5-d534-4f37-8260-5d5b415df0ff\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") "
Oct 31 13:16:14.711590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711386    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-credential-socket\") pod \"9206bfbe-392b-4a0e-972e-618cbafa13f9\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") "
Oct 31 13:16:14.711590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711454    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-token\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-token\") pod \"9206bfbe-392b-4a0e-972e-618cbafa13f9\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") "
Oct 31 13:16:14.711590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711491    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-podinfo\") pod \"9206bfbe-392b-4a0e-972e-618cbafa13f9\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") "
Oct 31 13:16:14.711590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711508    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9206bfbe-392b-4a0e-972e-618cbafa13f9-istiod-ca-cert" pod "9206bfbe-392b-4a0e-972e-618cbafa13f9" (UID: "9206bfbe-392b-4a0e-972e-618cbafa13f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:14.711590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711533    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-certs\") pod \"27fe87e5-d534-4f37-8260-5d5b415df0ff\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") "
Oct 31 13:16:14.711590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711586    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-credential-socket\") pod \"27fe87e5-d534-4f37-8260-5d5b415df0ff\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") "
Oct 31 13:16:14.712084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711583    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27fe87e5-d534-4f37-8260-5d5b415df0ff-istiod-ca-cert" pod "27fe87e5-d534-4f37-8260-5d5b415df0ff" (UID: "27fe87e5-d534-4f37-8260-5d5b415df0ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:14.712084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711672    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-data\") pod \"9206bfbe-392b-4a0e-972e-618cbafa13f9\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") "
Oct 31 13:16:14.712084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711727    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-socket\") pod \"9206bfbe-392b-4a0e-972e-618cbafa13f9\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") "
Oct 31 13:16:14.712084 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712045    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-socket\") pod \"27fe87e5-d534-4f37-8260-5d5b415df0ff\" (UID: \"27fe87e5-d534-4f37-8260-5d5b415df0ff\") "
Oct 31 13:16:14.712287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712115    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-envoy\") pod \"9206bfbe-392b-4a0e-972e-618cbafa13f9\" (UID: \"9206bfbe-392b-4a0e-972e-618cbafa13f9\") "
Oct 31 13:16:14.713025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712611    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.713025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712638    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.713025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712653    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9206bfbe-392b-4a0e-972e-618cbafa13f9-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.713025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712670    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27fe87e5-d534-4f37-8260-5d5b415df0ff-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.713025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712684    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2b6j\" (UniqueName: \"kubernetes.io/projected/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-kube-api-access-l2b6j\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.713025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712720    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.714498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.711803    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-credential-socket" pod "27fe87e5-d534-4f37-8260-5d5b415df0ff" (UID: "27fe87e5-d534-4f37-8260-5d5b415df0ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.714498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.714485    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-envoy" pod "27fe87e5-d534-4f37-8260-5d5b415df0ff" (UID: "27fe87e5-d534-4f37-8260-5d5b415df0ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.714676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712021    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-credential-socket" pod "9206bfbe-392b-4a0e-972e-618cbafa13f9" (UID: "9206bfbe-392b-4a0e-972e-618cbafa13f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.714676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712554    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-certs" pod "9206bfbe-392b-4a0e-972e-618cbafa13f9" (UID: "9206bfbe-392b-4a0e-972e-618cbafa13f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.714676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712754    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-socket" pod "9206bfbe-392b-4a0e-972e-618cbafa13f9" (UID: "9206bfbe-392b-4a0e-972e-618cbafa13f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.714676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.712939    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-data" pod "9206bfbe-392b-4a0e-972e-618cbafa13f9" (UID: "9206bfbe-392b-4a0e-972e-618cbafa13f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.714676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.714183    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-token" pod "27fe87e5-d534-4f37-8260-5d5b415df0ff" (UID: "27fe87e5-d534-4f37-8260-5d5b415df0ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:14.714676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.714458    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-podinfo" pod "27fe87e5-d534-4f37-8260-5d5b415df0ff" (UID: "27fe87e5-d534-4f37-8260-5d5b415df0ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:14.714676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.714582    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-kube-api-access-v5m94" pod "27fe87e5-d534-4f37-8260-5d5b415df0ff" (UID: "27fe87e5-d534-4f37-8260-5d5b415df0ff"). InnerVolumeSpecName "kube-api-access-v5m94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:14.714676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.714617    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-token" pod "9206bfbe-392b-4a0e-972e-618cbafa13f9" (UID: "9206bfbe-392b-4a0e-972e-618cbafa13f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:14.715048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.714782    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-data" pod "27fe87e5-d534-4f37-8260-5d5b415df0ff" (UID: "27fe87e5-d534-4f37-8260-5d5b415df0ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.715048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.714821    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-certs" pod "27fe87e5-d534-4f37-8260-5d5b415df0ff" (UID: "27fe87e5-d534-4f37-8260-5d5b415df0ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.715048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.714830    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-socket" pod "27fe87e5-d534-4f37-8260-5d5b415df0ff" (UID: "27fe87e5-d534-4f37-8260-5d5b415df0ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.715296 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.715271    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-kube-api-access-qc7km" pod "9206bfbe-392b-4a0e-972e-618cbafa13f9" (UID: "9206bfbe-392b-4a0e-972e-618cbafa13f9"). InnerVolumeSpecName "kube-api-access-qc7km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:14.716097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.716074    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-envoy" pod "9206bfbe-392b-4a0e-972e-618cbafa13f9" (UID: "9206bfbe-392b-4a0e-972e-618cbafa13f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:14.717047 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.717029    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-podinfo" pod "9206bfbe-392b-4a0e-972e-618cbafa13f9" (UID: "9206bfbe-392b-4a0e-972e-618cbafa13f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:14.808617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.808524    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:16:14.808617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.808593    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:16:14.815195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815164    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc7km\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-kube-api-access-qc7km\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815393 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815376    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815497 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815482    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5m94\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-kube-api-access-v5m94\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815582 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815571    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815682 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815662    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815686    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815715    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815730    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815743    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815760    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815774    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.815793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815787    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.816055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815802    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.816055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815815    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.816055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815829    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27fe87e5-d534-4f37-8260-5d5b415df0ff-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.816055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.815842    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9206bfbe-392b-4a0e-972e-618cbafa13f9-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:14.860412 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.860370    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"]
Oct 31 13:16:14.870298 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:14.870251    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-nfqls"]
Oct 31 13:16:15.383929 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.383889    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786" podUID="b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:16:15.384125 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.383889    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x" podUID="a6893107-8ce0-40a6-846e-a7eecfdbe6f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 13:16:15.552121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.552087    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"
Oct 31 13:16:15.552121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.552093    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr" event={"ID":"9206bfbe-392b-4a0e-972e-618cbafa13f9","Type":"ContainerDied","Data":"b6ab8e1bbb3e3a50d6c27b004207fa228d71a9b151694aa887b11a7838302683"}
Oct 31 13:16:15.552383 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.552204    5787 scope.go:122] "RemoveContainer" containerID="142c71f627c40f94389a904d16edd3f684c7ee28e9dfa1d2363b05e90d112fa9"
Oct 31 13:16:15.554077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.554000    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"
Oct 31 13:16:15.554200 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.554000    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g" event={"ID":"27fe87e5-d534-4f37-8260-5d5b415df0ff","Type":"ContainerDied","Data":"4addc61a5efad3af12e50b1d709423daefbe972ffd3d34837c39a248634f5f52"}
Oct 31 13:16:15.571523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.571500    5787 scope.go:122] "RemoveContainer" containerID="4dd6ae5374e13bbc98fff885b4642abd98e04590a1b2d2b56c3fae2badc6b98e"
Oct 31 13:16:15.591085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.591057    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"]
Oct 31 13:16:15.599385 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.599358    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-rqvvr"]
Oct 31 13:16:15.611649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.611617    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"]
Oct 31 13:16:15.618959 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.618933    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2a262a-openshift-default-5d9b94b5cc-h8d8g"]
Oct 31 13:16:15.979348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.979308    5787 patch_prober.go:29] interesting pod/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:16:15.979767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:15.979378    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:16:16.706458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:16.706427    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" path="/var/lib/kubelet/pods/27fe87e5-d534-4f37-8260-5d5b415df0ff/volumes"
Oct 31 13:16:16.706909 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:16.706895    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" path="/var/lib/kubelet/pods/4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b/volumes"
Oct 31 13:16:16.707304 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:16.707292    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" path="/var/lib/kubelet/pods/9206bfbe-392b-4a0e-972e-618cbafa13f9/volumes"
Oct 31 13:16:17.308820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.306963    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:16:17.356215 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.355214    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-plugins-config-volume\") pod \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\" (UID: \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\") "
Oct 31 13:16:17.356215 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.355438    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-kube-api-access-qqq7h\" (UniqueName: \"kubernetes.io/projected/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-kube-api-access-qqq7h\") pod \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\" (UID: \"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75\") "
Oct 31 13:16:17.357008 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.356960    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-plugins-config-volume" pod "b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" (UID: "b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:17.370824 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.370754    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-kube-api-access-qqq7h" pod "b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" (UID: "b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75"). InnerVolumeSpecName "kube-api-access-qqq7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:17.431168 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.430817    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:16:17.458464 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.458434    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqq7h\" (UniqueName: \"kubernetes.io/projected/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-kube-api-access-qqq7h\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.458749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.458695    5787 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75-plugins-config-volume\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.553895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.553865    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:16:17.557285 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.557261    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:16:17.559628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.559610    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-kube-api-access-vjbvw\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-kube-api-access-vjbvw\") pod \"07d9a104-017d-48ce-8831-5d9e0635b637\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") "
Oct 31 13:16:17.559737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.559641    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d9a104-017d-48ce-8831-5d9e0635b637-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d9a104-017d-48ce-8831-5d9e0635b637-istiod-ca-cert\") pod \"07d9a104-017d-48ce-8831-5d9e0635b637\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") "
Oct 31 13:16:17.559737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.559684    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-istio-token\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-istio-token\") pod \"07d9a104-017d-48ce-8831-5d9e0635b637\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") "
Oct 31 13:16:17.559737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.559722    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07d9a104-017d-48ce-8831-5d9e0635b637-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d9a104-017d-48ce-8831-5d9e0635b637-istio-podinfo\") pod \"07d9a104-017d-48ce-8831-5d9e0635b637\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") "
Oct 31 13:16:17.559903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.559750    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-socket\") pod \"07d9a104-017d-48ce-8831-5d9e0635b637\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") "
Oct 31 13:16:17.559903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.559827    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-envoy\") pod \"07d9a104-017d-48ce-8831-5d9e0635b637\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") "
Oct 31 13:16:17.559903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.559854    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-credential-socket\") pod \"07d9a104-017d-48ce-8831-5d9e0635b637\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") "
Oct 31 13:16:17.559903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.559896    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-data\") pod \"07d9a104-017d-48ce-8831-5d9e0635b637\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") "
Oct 31 13:16:17.560106 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.559930    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-certs\") pod \"07d9a104-017d-48ce-8831-5d9e0635b637\" (UID: \"07d9a104-017d-48ce-8831-5d9e0635b637\") "
Oct 31 13:16:17.560769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.560427    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-socket" pod "07d9a104-017d-48ce-8831-5d9e0635b637" (UID: "07d9a104-017d-48ce-8831-5d9e0635b637"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.560769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.560445    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-certs" pod "07d9a104-017d-48ce-8831-5d9e0635b637" (UID: "07d9a104-017d-48ce-8831-5d9e0635b637"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.560769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.560645    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d9a104-017d-48ce-8831-5d9e0635b637-istiod-ca-cert" pod "07d9a104-017d-48ce-8831-5d9e0635b637" (UID: "07d9a104-017d-48ce-8831-5d9e0635b637"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:17.560769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.560647    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-credential-socket" pod "07d9a104-017d-48ce-8831-5d9e0635b637" (UID: "07d9a104-017d-48ce-8831-5d9e0635b637"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.560769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.560676    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-data" pod "07d9a104-017d-48ce-8831-5d9e0635b637" (UID: "07d9a104-017d-48ce-8831-5d9e0635b637"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.563808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.563756    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-istio-token" pod "07d9a104-017d-48ce-8831-5d9e0635b637" (UID: "07d9a104-017d-48ce-8831-5d9e0635b637"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:17.564098 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.564067    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-kube-api-access-vjbvw" pod "07d9a104-017d-48ce-8831-5d9e0635b637" (UID: "07d9a104-017d-48ce-8831-5d9e0635b637"). InnerVolumeSpecName "kube-api-access-vjbvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:17.565345 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.564640    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07d9a104-017d-48ce-8831-5d9e0635b637-istio-podinfo" pod "07d9a104-017d-48ce-8831-5d9e0635b637" (UID: "07d9a104-017d-48ce-8831-5d9e0635b637"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:17.565345 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.564350    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-envoy" pod "07d9a104-017d-48ce-8831-5d9e0635b637" (UID: "07d9a104-017d-48ce-8831-5d9e0635b637"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.602588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.602524    5787 generic.go:356] "Generic (PLEG): container finished" podID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerID="49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16" exitCode=0
Oct 31 13:16:17.602914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.602888    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" event={"ID":"c6f8f023-3eb8-429c-8a06-d9a39fe51981","Type":"ContainerDied","Data":"49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16"}
Oct 31 13:16:17.603088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.603071    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp" event={"ID":"c6f8f023-3eb8-429c-8a06-d9a39fe51981","Type":"ContainerDied","Data":"a18e3946a4125b231270b497c9970fdbfdb2450a72409a6fd204552b80d52814"}
Oct 31 13:16:17.605441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.605419    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" event={"ID":"abd55913-612c-4bc3-a067-d11002f9f182","Type":"ContainerDied","Data":"f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9"}
Oct 31 13:16:17.605570 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.603301    5787 scope.go:122] "RemoveContainer" containerID="49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16"
Oct 31 13:16:17.605690 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.605250    5787 generic.go:356] "Generic (PLEG): container finished" podID="abd55913-612c-4bc3-a067-d11002f9f182" containerID="f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9" exitCode=0
Oct 31 13:16:17.605795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.605299    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"
Oct 31 13:16:17.608609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.603281    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"
Oct 31 13:16:17.608940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.605854    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47" event={"ID":"abd55913-612c-4bc3-a067-d11002f9f182","Type":"ContainerDied","Data":"dcf6a05d732117ae0ab1d72e90e76520a10d8e0e5e5d9a04737b4b7b9a2c422f"}
Oct 31 13:16:17.609073 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.609059    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786" event={"ID":"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75","Type":"ContainerDied","Data":"7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236"}
Oct 31 13:16:17.609171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.608324    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"
Oct 31 13:16:17.611005 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.608273    5787 generic.go:356] "Generic (PLEG): container finished" podID="b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" containerID="7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236" exitCode=0
Oct 31 13:16:17.611005 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.610478    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786" event={"ID":"b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75","Type":"ContainerDied","Data":"f7f58a190fa89b3ee6f275a5f46b46e2b27d25db1296724b074237dc7dd3bbbd"}
Oct 31 13:16:17.617733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.614783    5787 generic.go:356] "Generic (PLEG): container finished" podID="07d9a104-017d-48ce-8831-5d9e0635b637" containerID="6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e" exitCode=0
Oct 31 13:16:17.617733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.614864    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"
Oct 31 13:16:17.618325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.618221    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" event={"ID":"07d9a104-017d-48ce-8831-5d9e0635b637","Type":"ContainerDied","Data":"6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e"}
Oct 31 13:16:17.618325 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.618287    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c" event={"ID":"07d9a104-017d-48ce-8831-5d9e0635b637","Type":"ContainerDied","Data":"7b91d7962b9ec9aadfb81f78e640002e6d8e1d3d796deb220abf394645e37836"}
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.661595    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd55913-612c-4bc3-a067-d11002f9f182-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd55913-612c-4bc3-a067-d11002f9f182-istiod-ca-cert\") pod \"abd55913-612c-4bc3-a067-d11002f9f182\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.661651    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-data\") pod \"abd55913-612c-4bc3-a067-d11002f9f182\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.661719    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-socket\") pod \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.661757    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-kube-api-access-2kz28\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-kube-api-access-2kz28\") pod \"abd55913-612c-4bc3-a067-d11002f9f182\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.661804    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-envoy\") pod \"abd55913-612c-4bc3-a067-d11002f9f182\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.661863    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-kube-api-access-qrk5g\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-kube-api-access-qrk5g\") pod \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.661892    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-data\") pod \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.661931    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-token\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-token\") pod \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.661993    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-podinfo\") pod \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.662045    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-credential-socket\") pod \"abd55913-612c-4bc3-a067-d11002f9f182\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.662091    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abd55913-612c-4bc3-a067-d11002f9f182-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd55913-612c-4bc3-a067-d11002f9f182-istio-podinfo\") pod \"abd55913-612c-4bc3-a067-d11002f9f182\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.662122    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-certs\") pod \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") "
Oct 31 13:16:17.662470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.662153    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-envoy\") pod \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") "
Oct 31 13:16:17.663209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.663180    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-credential-socket" pod "c6f8f023-3eb8-429c-8a06-d9a39fe51981" (UID: "c6f8f023-3eb8-429c-8a06-d9a39fe51981"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.663492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.663473    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd55913-612c-4bc3-a067-d11002f9f182-istiod-ca-cert" pod "abd55913-612c-4bc3-a067-d11002f9f182" (UID: "abd55913-612c-4bc3-a067-d11002f9f182"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:17.663699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.663681    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-data" pod "abd55913-612c-4bc3-a067-d11002f9f182" (UID: "abd55913-612c-4bc3-a067-d11002f9f182"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.663909 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.663891    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-socket" pod "c6f8f023-3eb8-429c-8a06-d9a39fe51981" (UID: "c6f8f023-3eb8-429c-8a06-d9a39fe51981"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.662199    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-credential-socket\") pod \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") "
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.664810    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-certs\") pod \"abd55913-612c-4bc3-a067-d11002f9f182\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") "
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.664851    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-istio-token\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-istio-token\") pod \"abd55913-612c-4bc3-a067-d11002f9f182\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") "
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.664896    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-socket\") pod \"abd55913-612c-4bc3-a067-d11002f9f182\" (UID: \"abd55913-612c-4bc3-a067-d11002f9f182\") "
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.664963    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istiod-ca-cert\") pod \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\" (UID: \"c6f8f023-3eb8-429c-8a06-d9a39fe51981\") "
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665473    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjbvw\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-kube-api-access-vjbvw\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665496    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07d9a104-017d-48ce-8831-5d9e0635b637-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665510    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07d9a104-017d-48ce-8831-5d9e0635b637-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665523    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07d9a104-017d-48ce-8831-5d9e0635b637-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665536    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665560    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665572    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665586    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665599    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665612    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07d9a104-017d-48ce-8831-5d9e0635b637-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665626    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd55913-612c-4bc3-a067-d11002f9f182-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665641    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665653    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.665835 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.665808    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"]
Oct 31 13:16:17.666957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.666858    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istiod-ca-cert" pod "c6f8f023-3eb8-429c-8a06-d9a39fe51981" (UID: "c6f8f023-3eb8-429c-8a06-d9a39fe51981"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:17.667476 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.667216    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-certs" pod "abd55913-612c-4bc3-a067-d11002f9f182" (UID: "abd55913-612c-4bc3-a067-d11002f9f182"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.669199 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.669180    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-socket" pod "abd55913-612c-4bc3-a067-d11002f9f182" (UID: "abd55913-612c-4bc3-a067-d11002f9f182"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.676254 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.670446    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-004f64-694cccdc6c-gn786"]
Oct 31 13:16:17.676555 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.671314    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-data" pod "c6f8f023-3eb8-429c-8a06-d9a39fe51981" (UID: "c6f8f023-3eb8-429c-8a06-d9a39fe51981"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.676684 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.672150    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-certs" pod "c6f8f023-3eb8-429c-8a06-d9a39fe51981" (UID: "c6f8f023-3eb8-429c-8a06-d9a39fe51981"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.676857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.674342    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-credential-socket" pod "abd55913-612c-4bc3-a067-d11002f9f182" (UID: "abd55913-612c-4bc3-a067-d11002f9f182"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.692073 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.691971    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-kube-api-access-2kz28" pod "abd55913-612c-4bc3-a067-d11002f9f182" (UID: "abd55913-612c-4bc3-a067-d11002f9f182"). InnerVolumeSpecName "kube-api-access-2kz28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:17.693670 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.693548    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-envoy" pod "c6f8f023-3eb8-429c-8a06-d9a39fe51981" (UID: "c6f8f023-3eb8-429c-8a06-d9a39fe51981"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.693670 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.693633    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abd55913-612c-4bc3-a067-d11002f9f182-istio-podinfo" pod "abd55913-612c-4bc3-a067-d11002f9f182" (UID: "abd55913-612c-4bc3-a067-d11002f9f182"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:17.696813 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.696777    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-kube-api-access-qrk5g" pod "c6f8f023-3eb8-429c-8a06-d9a39fe51981" (UID: "c6f8f023-3eb8-429c-8a06-d9a39fe51981"). InnerVolumeSpecName "kube-api-access-qrk5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:17.698873 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.698836    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-token" pod "c6f8f023-3eb8-429c-8a06-d9a39fe51981" (UID: "c6f8f023-3eb8-429c-8a06-d9a39fe51981"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:17.700298 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.700266    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-podinfo" pod "c6f8f023-3eb8-429c-8a06-d9a39fe51981" (UID: "c6f8f023-3eb8-429c-8a06-d9a39fe51981"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:17.701276 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.701254    5787 scope.go:122] "RemoveContainer" containerID="49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16"
Oct 31 13:16:17.701891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.701683    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-envoy" pod "abd55913-612c-4bc3-a067-d11002f9f182" (UID: "abd55913-612c-4bc3-a067-d11002f9f182"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:17.701891 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:17.701770    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16\": container with ID starting with 49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16 not found: ID does not exist" containerID="49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16"
Oct 31 13:16:17.701891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.701794    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16"} err="failed to get container status \"49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16\": rpc error: code = NotFound desc = could not find container \"49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16\": container with ID starting with 49806dc4be38fe5c579d46ef9f7b53e29d8e0ddd8d94c56fbc6a1d24f903ca16 not found: ID does not exist"
Oct 31 13:16:17.701891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.701811    5787 scope.go:122] "RemoveContainer" containerID="f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9"
Oct 31 13:16:17.708793 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.708739    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"]
Oct 31 13:16:17.713011 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.712991    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-f4s4c"]
Oct 31 13:16:17.715675 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.715648    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-istio-token" pod "abd55913-612c-4bc3-a067-d11002f9f182" (UID: "abd55913-612c-4bc3-a067-d11002f9f182"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:17.729924 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.729894    5787 scope.go:122] "RemoveContainer" containerID="f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9"
Oct 31 13:16:17.730443 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:17.730238    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9\": container with ID starting with f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9 not found: ID does not exist" containerID="f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9"
Oct 31 13:16:17.730443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.730266    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9"} err="failed to get container status \"f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9\": rpc error: code = NotFound desc = could not find container \"f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9\": container with ID starting with f8c77cf817ed29f67a84efc455990c5c0215e50223cb17e8e60973b5d1fc06d9 not found: ID does not exist"
Oct 31 13:16:17.730443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.730287    5787 scope.go:122] "RemoveContainer" containerID="7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236"
Oct 31 13:16:17.742648 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.742628    5787 scope.go:122] "RemoveContainer" containerID="7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236"
Oct 31 13:16:17.743026 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:17.743003    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236\": container with ID starting with 7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236 not found: ID does not exist" containerID="7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236"
Oct 31 13:16:17.743138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.743038    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236"} err="failed to get container status \"7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236\": rpc error: code = NotFound desc = could not find container \"7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236\": container with ID starting with 7363f1e1357b0811e1f2d16ede6f1e83f7655f4e6bc15c7be92c9088f9566236 not found: ID does not exist"
Oct 31 13:16:17.743138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.743065    5787 scope.go:122] "RemoveContainer" containerID="6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e"
Oct 31 13:16:17.753942 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.753919    5787 scope.go:122] "RemoveContainer" containerID="82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5"
Oct 31 13:16:17.764616 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.764581    5787 scope.go:122] "RemoveContainer" containerID="6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e"
Oct 31 13:16:17.764991 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:17.764968    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e\": container with ID starting with 6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e not found: ID does not exist" containerID="6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e"
Oct 31 13:16:17.765104 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.765000    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e"} err="failed to get container status \"6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e\": rpc error: code = NotFound desc = could not find container \"6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e\": container with ID starting with 6bffcaa779d6129a6ce56df504fe67a5603d907e15adbfc8dd5f7e7a5211647e not found: ID does not exist"
Oct 31 13:16:17.765104 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.765025    5787 scope.go:122] "RemoveContainer" containerID="82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5"
Oct 31 13:16:17.765313 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:17.765291    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5\": container with ID starting with 82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5 not found: ID does not exist" containerID="82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5"
Oct 31 13:16:17.765409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.765322    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5"} err="failed to get container status \"82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5\": rpc error: code = NotFound desc = could not find container \"82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5\": container with ID starting with 82b25073c6a534b963d0c031372e64b8373990e4bec06eceee502fb8d3b304b5 not found: ID does not exist"
Oct 31 13:16:17.767572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767547    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767577    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767593    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd55913-612c-4bc3-a067-d11002f9f182-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767605    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767618    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767630    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767644    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767655    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767668    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767680    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kz28\" (UniqueName: \"kubernetes.io/projected/abd55913-612c-4bc3-a067-d11002f9f182-kube-api-access-2kz28\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767693    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd55913-612c-4bc3-a067-d11002f9f182-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767722    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrk5g\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-kube-api-access-qrk5g\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767736    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:17.767886 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:17.767749    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f8f023-3eb8-429c-8a06-d9a39fe51981-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.429056 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.429012    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"]
Oct 31 13:16:18.446798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.446769    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-cbzpp"]
Oct 31 13:16:18.458412 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.458359    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"]
Oct 31 13:16:18.499797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.499677    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-004f64-openshift-default-95df6bdc-nst47"]
Oct 31 13:16:18.577773 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:18.574505    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:16:18.657244 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.657198    5787 generic.go:356] "Generic (PLEG): container finished" podID="b889c3d9-985e-45d9-941e-539936378181" containerID="f067cb8e433ecf1b49d85f38799fa2964bed65959a8c234f03beb144a744df1f" exitCode=0
Oct 31 13:16:18.657418 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.657307    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" event={"ID":"b889c3d9-985e-45d9-941e-539936378181","Type":"ContainerDied","Data":"f067cb8e433ecf1b49d85f38799fa2964bed65959a8c234f03beb144a744df1f"}
Oct 31 13:16:18.659773 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.659740    5787 generic.go:356] "Generic (PLEG): container finished" podID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerID="f1a4b404ccdaca53844fafb7a55b0fbbdfea74b1e136bc2fe97087b9742fd770" exitCode=0
Oct 31 13:16:18.659955 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.659818    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" event={"ID":"8a1da508-b47c-4513-9584-a5b4771d4f5d","Type":"ContainerDied","Data":"f1a4b404ccdaca53844fafb7a55b0fbbdfea74b1e136bc2fe97087b9742fd770"}
Oct 31 13:16:18.664454 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.664423    5787 generic.go:356] "Generic (PLEG): container finished" podID="a6893107-8ce0-40a6-846e-a7eecfdbe6f5" containerID="e6eb4fb8f1b406c7f5f605af2bf7474635fb42f63b1f2316f4cfae7a4e6b7d90" exitCode=0
Oct 31 13:16:18.664570 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.664529    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x" event={"ID":"a6893107-8ce0-40a6-846e-a7eecfdbe6f5","Type":"ContainerDied","Data":"e6eb4fb8f1b406c7f5f605af2bf7474635fb42f63b1f2316f4cfae7a4e6b7d90"}
Oct 31 13:16:18.667449 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.666570    5787 generic.go:356] "Generic (PLEG): container finished" podID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerID="d8ef1b7043a1afc31c3ac72c40c3aa27744a7ea7276db0a8061c35bee2371ef5" exitCode=0
Oct 31 13:16:18.667651 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.667622    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" event={"ID":"94e7efa4-f7a0-4268-8356-91ef1caaf034","Type":"ContainerDied","Data":"d8ef1b7043a1afc31c3ac72c40c3aa27744a7ea7276db0a8061c35bee2371ef5"}
Oct 31 13:16:18.709118 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.709069    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" path="/var/lib/kubelet/pods/07d9a104-017d-48ce-8831-5d9e0635b637/volumes"
Oct 31 13:16:18.709837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.709813    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd55913-612c-4bc3-a067-d11002f9f182" path="/var/lib/kubelet/pods/abd55913-612c-4bc3-a067-d11002f9f182/volumes"
Oct 31 13:16:18.710378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.710355    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" path="/var/lib/kubelet/pods/b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75/volumes"
Oct 31 13:16:18.711441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.711420    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" path="/var/lib/kubelet/pods/c6f8f023-3eb8-429c-8a06-d9a39fe51981/volumes"
Oct 31 13:16:18.720382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.720354    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:16:18.774088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.774055    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:16:18.776078 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776047    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-socket\") pod \"8a1da508-b47c-4513-9584-a5b4771d4f5d\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") "
Oct 31 13:16:18.776220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776114    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-data\") pod \"8a1da508-b47c-4513-9584-a5b4771d4f5d\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") "
Oct 31 13:16:18.776220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776154    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a1da508-b47c-4513-9584-a5b4771d4f5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1da508-b47c-4513-9584-a5b4771d4f5d-istiod-ca-cert\") pod \"8a1da508-b47c-4513-9584-a5b4771d4f5d\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") "
Oct 31 13:16:18.776220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776188    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-certs\") pod \"8a1da508-b47c-4513-9584-a5b4771d4f5d\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") "
Oct 31 13:16:18.776220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776217    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-envoy\") pod \"8a1da508-b47c-4513-9584-a5b4771d4f5d\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") "
Oct 31 13:16:18.776429 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776249    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-token\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-token\") pod \"8a1da508-b47c-4513-9584-a5b4771d4f5d\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") "
Oct 31 13:16:18.776429 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776341    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-socket" pod "8a1da508-b47c-4513-9584-a5b4771d4f5d" (UID: "8a1da508-b47c-4513-9584-a5b4771d4f5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.776429 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776362    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-kube-api-access-gncn7\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-kube-api-access-gncn7\") pod \"8a1da508-b47c-4513-9584-a5b4771d4f5d\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") "
Oct 31 13:16:18.776585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776443    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-podinfo\") pod \"8a1da508-b47c-4513-9584-a5b4771d4f5d\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") "
Oct 31 13:16:18.776585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776526    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-credential-socket\") pod \"8a1da508-b47c-4513-9584-a5b4771d4f5d\" (UID: \"8a1da508-b47c-4513-9584-a5b4771d4f5d\") "
Oct 31 13:16:18.776681 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776639    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-data" pod "8a1da508-b47c-4513-9584-a5b4771d4f5d" (UID: "8a1da508-b47c-4513-9584-a5b4771d4f5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.776934 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776910    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-credential-socket" pod "8a1da508-b47c-4513-9584-a5b4771d4f5d" (UID: "8a1da508-b47c-4513-9584-a5b4771d4f5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.777098 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.776917    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1da508-b47c-4513-9584-a5b4771d4f5d-istiod-ca-cert" pod "8a1da508-b47c-4513-9584-a5b4771d4f5d" (UID: "8a1da508-b47c-4513-9584-a5b4771d4f5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:18.777285 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.777212    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-certs" pod "8a1da508-b47c-4513-9584-a5b4771d4f5d" (UID: "8a1da508-b47c-4513-9584-a5b4771d4f5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.777285 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.777026    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.777285 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.777269    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.779762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.779680    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-envoy" pod "8a1da508-b47c-4513-9584-a5b4771d4f5d" (UID: "8a1da508-b47c-4513-9584-a5b4771d4f5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.779762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.779732    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-token" pod "8a1da508-b47c-4513-9584-a5b4771d4f5d" (UID: "8a1da508-b47c-4513-9584-a5b4771d4f5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:18.779878 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.779839    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-podinfo" pod "8a1da508-b47c-4513-9584-a5b4771d4f5d" (UID: "8a1da508-b47c-4513-9584-a5b4771d4f5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:18.781397 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.781377    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-kube-api-access-gncn7" pod "8a1da508-b47c-4513-9584-a5b4771d4f5d" (UID: "8a1da508-b47c-4513-9584-a5b4771d4f5d"). InnerVolumeSpecName "kube-api-access-gncn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:18.831916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.831891    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:16:18.833940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.833919    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:16:18.877808 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.877778    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-kube-api-access-zrc5s\" (UniqueName: \"kubernetes.io/projected/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-kube-api-access-zrc5s\") pod \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\" (UID: \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\") "
Oct 31 13:16:18.877997 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.877838    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-plugins-config-volume\") pod \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\" (UID: \"a6893107-8ce0-40a6-846e-a7eecfdbe6f5\") "
Oct 31 13:16:18.878289 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.878263    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-plugins-config-volume" pod "a6893107-8ce0-40a6-846e-a7eecfdbe6f5" (UID: "a6893107-8ce0-40a6-846e-a7eecfdbe6f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:18.878378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.878338    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gncn7\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-kube-api-access-gncn7\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.878378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.878358    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.878378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.878369    5787 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-plugins-config-volume\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.878378 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.878377    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.878782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.878385    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a1da508-b47c-4513-9584-a5b4771d4f5d-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.878782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.878394    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.878782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.878402    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.878782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.878410    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a1da508-b47c-4513-9584-a5b4771d4f5d-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.880354 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.880326    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-kube-api-access-zrc5s" pod "a6893107-8ce0-40a6-846e-a7eecfdbe6f5" (UID: "a6893107-8ce0-40a6-846e-a7eecfdbe6f5"). InnerVolumeSpecName "kube-api-access-zrc5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:18.978977 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.978943    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-envoy\") pod \"b889c3d9-985e-45d9-941e-539936378181\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") "
Oct 31 13:16:18.979235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979041    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b889c3d9-985e-45d9-941e-539936378181-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b889c3d9-985e-45d9-941e-539936378181-istio-podinfo\") pod \"b889c3d9-985e-45d9-941e-539936378181\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") "
Oct 31 13:16:18.979235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979064    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-certs\") pod \"b889c3d9-985e-45d9-941e-539936378181\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") "
Oct 31 13:16:18.979235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979087    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-kube-api-access-jjb24\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-kube-api-access-jjb24\") pod \"b889c3d9-985e-45d9-941e-539936378181\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") "
Oct 31 13:16:18.979235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979116    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-certs\") pod \"94e7efa4-f7a0-4268-8356-91ef1caaf034\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") "
Oct 31 13:16:18.979235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979165    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-socket\") pod \"94e7efa4-f7a0-4268-8356-91ef1caaf034\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") "
Oct 31 13:16:18.979235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979206    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-istio-token\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-istio-token\") pod \"b889c3d9-985e-45d9-941e-539936378181\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979266    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-data\") pod \"b889c3d9-985e-45d9-941e-539936378181\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979301    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-kube-api-access-85kj2\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-kube-api-access-85kj2\") pod \"94e7efa4-f7a0-4268-8356-91ef1caaf034\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979329    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-envoy\") pod \"94e7efa4-f7a0-4268-8356-91ef1caaf034\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979370    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-credential-socket\") pod \"94e7efa4-f7a0-4268-8356-91ef1caaf034\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979370    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-certs" pod "b889c3d9-985e-45d9-941e-539936378181" (UID: "b889c3d9-985e-45d9-941e-539936378181"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979393    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-certs" pod "94e7efa4-f7a0-4268-8356-91ef1caaf034" (UID: "94e7efa4-f7a0-4268-8356-91ef1caaf034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979406    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-socket\") pod \"b889c3d9-985e-45d9-941e-539936378181\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979435    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e7efa4-f7a0-4268-8356-91ef1caaf034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e7efa4-f7a0-4268-8356-91ef1caaf034-istiod-ca-cert\") pod \"94e7efa4-f7a0-4268-8356-91ef1caaf034\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979466    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-podinfo\") pod \"94e7efa4-f7a0-4268-8356-91ef1caaf034\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979501    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-credential-socket\") pod \"b889c3d9-985e-45d9-941e-539936378181\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979532    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b889c3d9-985e-45d9-941e-539936378181-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b889c3d9-985e-45d9-941e-539936378181-istiod-ca-cert\") pod \"b889c3d9-985e-45d9-941e-539936378181\" (UID: \"b889c3d9-985e-45d9-941e-539936378181\") "
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979549    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-data" pod "b889c3d9-985e-45d9-941e-539936378181" (UID: "b889c3d9-985e-45d9-941e-539936378181"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.979552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979521    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-socket" pod "94e7efa4-f7a0-4268-8356-91ef1caaf034" (UID: "94e7efa4-f7a0-4268-8356-91ef1caaf034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.980451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979585    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-credential-socket" pod "94e7efa4-f7a0-4268-8356-91ef1caaf034" (UID: "94e7efa4-f7a0-4268-8356-91ef1caaf034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.980451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979817    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-credential-socket" pod "b889c3d9-985e-45d9-941e-539936378181" (UID: "b889c3d9-985e-45d9-941e-539936378181"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.980451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.979853    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-socket" pod "b889c3d9-985e-45d9-941e-539936378181" (UID: "b889c3d9-985e-45d9-941e-539936378181"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.980451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980244    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-data\") pod \"94e7efa4-f7a0-4268-8356-91ef1caaf034\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") "
Oct 31 13:16:18.980451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980294    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-token\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-token\") pod \"94e7efa4-f7a0-4268-8356-91ef1caaf034\" (UID: \"94e7efa4-f7a0-4268-8356-91ef1caaf034\") "
Oct 31 13:16:18.980451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980401    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b889c3d9-985e-45d9-941e-539936378181-istiod-ca-cert" pod "b889c3d9-985e-45d9-941e-539936378181" (UID: "b889c3d9-985e-45d9-941e-539936378181"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:18.980797 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980608    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-data" pod "94e7efa4-f7a0-4268-8356-91ef1caaf034" (UID: "94e7efa4-f7a0-4268-8356-91ef1caaf034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.980867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980812    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.980867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980816    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e7efa4-f7a0-4268-8356-91ef1caaf034-istiod-ca-cert" pod "94e7efa4-f7a0-4268-8356-91ef1caaf034" (UID: "94e7efa4-f7a0-4268-8356-91ef1caaf034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:18.980867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980831    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.980867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980852    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.981077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980867    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrc5s\" (UniqueName: \"kubernetes.io/projected/a6893107-8ce0-40a6-846e-a7eecfdbe6f5-kube-api-access-zrc5s\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.981077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980880    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.981077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980893    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.981077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980906    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.981077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980918    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.981077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980933    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b889c3d9-985e-45d9-941e-539936378181-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.981077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.980945    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:18.982013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.981974    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-envoy" pod "b889c3d9-985e-45d9-941e-539936378181" (UID: "b889c3d9-985e-45d9-941e-539936378181"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.982408 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.982384    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-envoy" pod "94e7efa4-f7a0-4268-8356-91ef1caaf034" (UID: "94e7efa4-f7a0-4268-8356-91ef1caaf034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:18.982698 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.982661    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-kube-api-access-jjb24" pod "b889c3d9-985e-45d9-941e-539936378181" (UID: "b889c3d9-985e-45d9-941e-539936378181"). InnerVolumeSpecName "kube-api-access-jjb24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:18.983031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.983006    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-istio-token" pod "b889c3d9-985e-45d9-941e-539936378181" (UID: "b889c3d9-985e-45d9-941e-539936378181"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:18.983097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.983051    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b889c3d9-985e-45d9-941e-539936378181-istio-podinfo" pod "b889c3d9-985e-45d9-941e-539936378181" (UID: "b889c3d9-985e-45d9-941e-539936378181"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:18.983190 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.983178    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-kube-api-access-85kj2" pod "94e7efa4-f7a0-4268-8356-91ef1caaf034" (UID: "94e7efa4-f7a0-4268-8356-91ef1caaf034"). InnerVolumeSpecName "kube-api-access-85kj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:18.983354 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.983335    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-podinfo" pod "94e7efa4-f7a0-4268-8356-91ef1caaf034" (UID: "94e7efa4-f7a0-4268-8356-91ef1caaf034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:18.983588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:18.983576    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-token" pod "94e7efa4-f7a0-4268-8356-91ef1caaf034" (UID: "94e7efa4-f7a0-4268-8356-91ef1caaf034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:19.082294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.082199    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b889c3d9-985e-45d9-941e-539936378181-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:19.082294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.082234    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b889c3d9-985e-45d9-941e-539936378181-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:19.082294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.082246    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjb24\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-kube-api-access-jjb24\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:19.082294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.082255    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b889c3d9-985e-45d9-941e-539936378181-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:19.082294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.082265    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85kj2\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-kube-api-access-85kj2\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:19.082294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.082279    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:19.082294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.082290    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e7efa4-f7a0-4268-8356-91ef1caaf034-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:19.082294 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.082299    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:19.082676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.082309    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e7efa4-f7a0-4268-8356-91ef1caaf034-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:19.680751 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.680720    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"
Oct 31 13:16:19.680751 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.680743    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x" event={"ID":"a6893107-8ce0-40a6-846e-a7eecfdbe6f5","Type":"ContainerDied","Data":"0f1d91241dd7ab65dec5fed68964337d96dbbe5ecfe757c523e758c4ec4554f5"}
Oct 31 13:16:19.681292 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.680810    5787 scope.go:122] "RemoveContainer" containerID="e6eb4fb8f1b406c7f5f605af2bf7474635fb42f63b1f2316f4cfae7a4e6b7d90"
Oct 31 13:16:19.682537 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.682520    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"
Oct 31 13:16:19.682735 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.682694    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb" event={"ID":"94e7efa4-f7a0-4268-8356-91ef1caaf034","Type":"ContainerDied","Data":"1743fb06bfed1565d8a97e4420eb9ca100279624e07b99b9b730be6df64bf4a0"}
Oct 31 13:16:19.684885 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.684862    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"
Oct 31 13:16:19.685014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.684887    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k" event={"ID":"b889c3d9-985e-45d9-941e-539936378181","Type":"ContainerDied","Data":"42dd12a775c3811e90bbd4eabd6245e69f0009a93362ba37372934cd79dc172f"}
Oct 31 13:16:19.686638 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.686613    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc" event={"ID":"8a1da508-b47c-4513-9584-a5b4771d4f5d","Type":"ContainerDied","Data":"61b1b9a8d25af21576adae1ada13fc520c27fe1477f7e796df7ffe0357c8e34c"}
Oct 31 13:16:19.686749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.686646    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"
Oct 31 13:16:19.695894 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.695867    5787 scope.go:122] "RemoveContainer" containerID="d8ef1b7043a1afc31c3ac72c40c3aa27744a7ea7276db0a8061c35bee2371ef5"
Oct 31 13:16:19.708268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.708248    5787 scope.go:122] "RemoveContainer" containerID="f067cb8e433ecf1b49d85f38799fa2964bed65959a8c234f03beb144a744df1f"
Oct 31 13:16:19.718680 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.718647    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"]
Oct 31 13:16:19.720239 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.720223    5787 scope.go:122] "RemoveContainer" containerID="f1a4b404ccdaca53844fafb7a55b0fbbdfea74b1e136bc2fe97087b9742fd770"
Oct 31 13:16:19.723025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.723002    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-nmbpb"]
Oct 31 13:16:19.738500 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.738473    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"]
Oct 31 13:16:19.742803 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.742779    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-004f64-6df99d7697-vlx8x"]
Oct 31 13:16:19.758126 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.758097    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"]
Oct 31 13:16:19.764093 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.764070    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-cz29k"]
Oct 31 13:16:19.777243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.777217    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"]
Oct 31 13:16:19.784075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:19.784051    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-426787-openshift-default-5ffc64c5c7-c6qnc"]
Oct 31 13:16:20.705543 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:20.705513    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" path="/var/lib/kubelet/pods/8a1da508-b47c-4513-9584-a5b4771d4f5d/volumes"
Oct 31 13:16:20.705982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:20.705968    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" path="/var/lib/kubelet/pods/94e7efa4-f7a0-4268-8356-91ef1caaf034/volumes"
Oct 31 13:16:20.706299 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:20.706288    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6893107-8ce0-40a6-846e-a7eecfdbe6f5" path="/var/lib/kubelet/pods/a6893107-8ce0-40a6-846e-a7eecfdbe6f5/volumes"
Oct 31 13:16:20.707015 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:20.706999    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b889c3d9-985e-45d9-941e-539936378181" path="/var/lib/kubelet/pods/b889c3d9-985e-45d9-941e-539936378181/volumes"
Oct 31 13:16:26.171463 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.171424    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"]
Oct 31 13:16:26.172178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.171757    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" containerID="cri-o://932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37" gracePeriod=30
Oct 31 13:16:26.192752 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.189629    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"]
Oct 31 13:16:26.192752 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.190039    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" containerID="cri-o://3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db" gracePeriod=30
Oct 31 13:16:26.198048 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.198010    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"]
Oct 31 13:16:26.198267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.198247    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy" containerID="cri-o://82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b" gracePeriod=30
Oct 31 13:16:26.203242 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.203215    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"]
Oct 31 13:16:26.203411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.203391    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy" containerID="cri-o://c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6" gracePeriod=30
Oct 31 13:16:26.238825 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.238778    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:16:26.239438 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:26.238855    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:16:31.074915 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.074872    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 13:16:31.075385 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.074961    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 13:16:31.264969 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.264932    5787 patch_prober.go:29] interesting pod/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Oct 31 13:16:31.265112 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.265011    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Oct 31 13:16:31.567295 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.567266    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:16:31.592898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.592810    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:16:31.655439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.655405    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-socket\") pod \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") "
Oct 31 13:16:31.655675 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.655659    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-credential-socket\") pod \"b687cd60-1308-468e-a24f-674c6cef2f8a\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") "
Oct 31 13:16:31.655834 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.655819    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-podinfo\") pod \"b687cd60-1308-468e-a24f-674c6cef2f8a\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") "
Oct 31 13:16:31.655945 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.655932    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-credential-socket\") pod \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") "
Oct 31 13:16:31.656054 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.656042    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-certs\") pod \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") "
Oct 31 13:16:31.656164 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.656151    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b687cd60-1308-468e-a24f-674c6cef2f8a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b687cd60-1308-468e-a24f-674c6cef2f8a-istiod-ca-cert\") pod \"b687cd60-1308-468e-a24f-674c6cef2f8a\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") "
Oct 31 13:16:31.656273 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.656261    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-socket\") pod \"b687cd60-1308-468e-a24f-674c6cef2f8a\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") "
Oct 31 13:16:31.657868 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.657668    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-podinfo\") pod \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") "
Oct 31 13:16:31.658451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.658421    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-envoy\") pod \"b687cd60-1308-468e-a24f-674c6cef2f8a\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") "
Oct 31 13:16:31.661173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.658631    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istiod-ca-cert\") pod \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") "
Oct 31 13:16:31.661367 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.661336    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-data\") pod \"b687cd60-1308-468e-a24f-674c6cef2f8a\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") "
Oct 31 13:16:31.661528 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.661504    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-kube-api-access-nfl2f\") pod \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") "
Oct 31 13:16:31.663386 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.662196    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b687cd60-1308-468e-a24f-674c6cef2f8a-istiod-ca-cert" pod "b687cd60-1308-468e-a24f-674c6cef2f8a" (UID: "b687cd60-1308-468e-a24f-674c6cef2f8a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:31.663386 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.662690    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-socket" pod "bc4b5f7e-1bed-46dd-b27f-e89f250107ec" (UID: "bc4b5f7e-1bed-46dd-b27f-e89f250107ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.663386 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.662991    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-credential-socket" pod "b687cd60-1308-468e-a24f-674c6cef2f8a" (UID: "b687cd60-1308-468e-a24f-674c6cef2f8a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.665642 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.665442    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-socket" pod "b687cd60-1308-468e-a24f-674c6cef2f8a" (UID: "b687cd60-1308-468e-a24f-674c6cef2f8a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.666011 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.665796    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-credential-socket" pod "bc4b5f7e-1bed-46dd-b27f-e89f250107ec" (UID: "bc4b5f7e-1bed-46dd-b27f-e89f250107ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.666422 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.666136    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-certs" pod "bc4b5f7e-1bed-46dd-b27f-e89f250107ec" (UID: "bc4b5f7e-1bed-46dd-b27f-e89f250107ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.666422 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.666201    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-podinfo" pod "b687cd60-1308-468e-a24f-674c6cef2f8a" (UID: "b687cd60-1308-468e-a24f-674c6cef2f8a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:31.666562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.666536    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istiod-ca-cert" pod "bc4b5f7e-1bed-46dd-b27f-e89f250107ec" (UID: "bc4b5f7e-1bed-46dd-b27f-e89f250107ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.666860    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-data" pod "b687cd60-1308-468e-a24f-674c6cef2f8a" (UID: "b687cd60-1308-468e-a24f-674c6cef2f8a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.667796    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-token\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-token\") pod \"b687cd60-1308-468e-a24f-674c6cef2f8a\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") "
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.667862    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-token\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-token\") pod \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") "
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.668026    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-envoy\") pod \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") "
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.668067    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-data\") pod \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\" (UID: \"bc4b5f7e-1bed-46dd-b27f-e89f250107ec\") "
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.668128    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-certs\") pod \"b687cd60-1308-468e-a24f-674c6cef2f8a\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") "
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.668177    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-kube-api-access-xvs5w\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-kube-api-access-xvs5w\") pod \"b687cd60-1308-468e-a24f-674c6cef2f8a\" (UID: \"b687cd60-1308-468e-a24f-674c6cef2f8a\") "
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.668926    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.668962    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.668976    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.668987    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.669014    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.669043    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b687cd60-1308-468e-a24f-674c6cef2f8a-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.669055    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.669069    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.669082    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.669693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.669451    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-data" pod "bc4b5f7e-1bed-46dd-b27f-e89f250107ec" (UID: "bc4b5f7e-1bed-46dd-b27f-e89f250107ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.673787 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.671230    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-certs" pod "b687cd60-1308-468e-a24f-674c6cef2f8a" (UID: "b687cd60-1308-468e-a24f-674c6cef2f8a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.693151 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.687881    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:16:31.693151 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.690646    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-kube-api-access-nfl2f" pod "bc4b5f7e-1bed-46dd-b27f-e89f250107ec" (UID: "bc4b5f7e-1bed-46dd-b27f-e89f250107ec"). InnerVolumeSpecName "kube-api-access-nfl2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:31.693151 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.692860    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-token" pod "b687cd60-1308-468e-a24f-674c6cef2f8a" (UID: "b687cd60-1308-468e-a24f-674c6cef2f8a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:31.696263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.693880    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-kube-api-access-xvs5w" pod "b687cd60-1308-468e-a24f-674c6cef2f8a" (UID: "b687cd60-1308-468e-a24f-674c6cef2f8a"). InnerVolumeSpecName "kube-api-access-xvs5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:31.696263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.694024    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-podinfo" pod "bc4b5f7e-1bed-46dd-b27f-e89f250107ec" (UID: "bc4b5f7e-1bed-46dd-b27f-e89f250107ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:31.696263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.694047    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-token" pod "bc4b5f7e-1bed-46dd-b27f-e89f250107ec" (UID: "bc4b5f7e-1bed-46dd-b27f-e89f250107ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:31.696263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.694339    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:16:31.696263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.695669    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-envoy" pod "b687cd60-1308-468e-a24f-674c6cef2f8a" (UID: "b687cd60-1308-468e-a24f-674c6cef2f8a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.696843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.696805    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-envoy" pod "bc4b5f7e-1bed-46dd-b27f-e89f250107ec" (UID: "bc4b5f7e-1bed-46dd-b27f-e89f250107ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.771722 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.771664    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-istio-token\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-istio-token\") pod \"17041e62-9718-42d3-9481-30851a9c7db0\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") "
Oct 31 13:16:31.772026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.772003    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-socket\") pod \"9f71d479-bb83-40b8-a668-9865152cf25d\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") "
Oct 31 13:16:31.772203 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.772185    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-socket\") pod \"17041e62-9718-42d3-9481-30851a9c7db0\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") "
Oct 31 13:16:31.772348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.772332    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17041e62-9718-42d3-9481-30851a9c7db0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17041e62-9718-42d3-9481-30851a9c7db0-istio-podinfo\") pod \"17041e62-9718-42d3-9481-30851a9c7db0\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") "
Oct 31 13:16:31.772509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.772491    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-envoy\") pod \"17041e62-9718-42d3-9481-30851a9c7db0\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") "
Oct 31 13:16:31.772638 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.772623    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-credential-socket\") pod \"9f71d479-bb83-40b8-a668-9865152cf25d\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") "
Oct 31 13:16:31.772812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.772795    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-kube-api-access-q4662\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-kube-api-access-q4662\") pod \"9f71d479-bb83-40b8-a668-9865152cf25d\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") "
Oct 31 13:16:31.772995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.772979    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-certs\") pod \"9f71d479-bb83-40b8-a668-9865152cf25d\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") "
Oct 31 13:16:31.773130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.773114    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f71d479-bb83-40b8-a668-9865152cf25d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f71d479-bb83-40b8-a668-9865152cf25d-istiod-ca-cert\") pod \"9f71d479-bb83-40b8-a668-9865152cf25d\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") "
Oct 31 13:16:31.773253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.773239    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-istio-token\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-istio-token\") pod \"9f71d479-bb83-40b8-a668-9865152cf25d\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") "
Oct 31 13:16:31.773382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.773367    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-data\") pod \"17041e62-9718-42d3-9481-30851a9c7db0\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") "
Oct 31 13:16:31.773498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.773484    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-data\") pod \"9f71d479-bb83-40b8-a668-9865152cf25d\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") "
Oct 31 13:16:31.773646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.773628    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-certs\") pod \"17041e62-9718-42d3-9481-30851a9c7db0\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") "
Oct 31 13:16:31.773860 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.773842    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17041e62-9718-42d3-9481-30851a9c7db0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17041e62-9718-42d3-9481-30851a9c7db0-istiod-ca-cert\") pod \"17041e62-9718-42d3-9481-30851a9c7db0\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") "
Oct 31 13:16:31.774046 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.774028    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-credential-socket\") pod \"17041e62-9718-42d3-9481-30851a9c7db0\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") "
Oct 31 13:16:31.775572 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.775551    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-envoy\") pod \"9f71d479-bb83-40b8-a668-9865152cf25d\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") "
Oct 31 13:16:31.775950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.775912    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-kube-api-access-g94kw\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-kube-api-access-g94kw\") pod \"17041e62-9718-42d3-9481-30851a9c7db0\" (UID: \"17041e62-9718-42d3-9481-30851a9c7db0\") "
Oct 31 13:16:31.776150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.776132    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9f71d479-bb83-40b8-a668-9865152cf25d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f71d479-bb83-40b8-a668-9865152cf25d-istio-podinfo\") pod \"9f71d479-bb83-40b8-a668-9865152cf25d\" (UID: \"9f71d479-bb83-40b8-a668-9865152cf25d\") "
Oct 31 13:16:31.777147 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.777126    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.777326 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.777310    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.777488 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.777472    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfl2f\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-kube-api-access-nfl2f\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.777679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.777645    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.777866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.777850    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.777981 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.777968    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.778080 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.778068    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc4b5f7e-1bed-46dd-b27f-e89f250107ec-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.778184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.778171    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b687cd60-1308-468e-a24f-674c6cef2f8a-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.778277 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.778264    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvs5w\" (UniqueName: \"kubernetes.io/projected/b687cd60-1308-468e-a24f-674c6cef2f8a-kube-api-access-xvs5w\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.778417 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.778102    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f71d479-bb83-40b8-a668-9865152cf25d-istiod-ca-cert" pod "9f71d479-bb83-40b8-a668-9865152cf25d" (UID: "9f71d479-bb83-40b8-a668-9865152cf25d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:31.780165 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.780141    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-socket" pod "17041e62-9718-42d3-9481-30851a9c7db0" (UID: "17041e62-9718-42d3-9481-30851a9c7db0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.788512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.788458    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-data" pod "17041e62-9718-42d3-9481-30851a9c7db0" (UID: "17041e62-9718-42d3-9481-30851a9c7db0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.789447 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.788641    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-socket" pod "9f71d479-bb83-40b8-a668-9865152cf25d" (UID: "9f71d479-bb83-40b8-a668-9865152cf25d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.790095 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.790069    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-certs" pod "17041e62-9718-42d3-9481-30851a9c7db0" (UID: "17041e62-9718-42d3-9481-30851a9c7db0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.790542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.790516    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17041e62-9718-42d3-9481-30851a9c7db0-istiod-ca-cert" pod "17041e62-9718-42d3-9481-30851a9c7db0" (UID: "17041e62-9718-42d3-9481-30851a9c7db0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:31.790949 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.790918    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-credential-socket" pod "17041e62-9718-42d3-9481-30851a9c7db0" (UID: "17041e62-9718-42d3-9481-30851a9c7db0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.791690 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.791662    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-data" pod "9f71d479-bb83-40b8-a668-9865152cf25d" (UID: "9f71d479-bb83-40b8-a668-9865152cf25d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.800447 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.793062    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-envoy" pod "17041e62-9718-42d3-9481-30851a9c7db0" (UID: "17041e62-9718-42d3-9481-30851a9c7db0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.800668 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.796260    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-credential-socket" pod "9f71d479-bb83-40b8-a668-9865152cf25d" (UID: "9f71d479-bb83-40b8-a668-9865152cf25d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.800822 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.799565    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-kube-api-access-q4662" pod "9f71d479-bb83-40b8-a668-9865152cf25d" (UID: "9f71d479-bb83-40b8-a668-9865152cf25d"). InnerVolumeSpecName "kube-api-access-q4662". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:31.804288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.804145    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f71d479-bb83-40b8-a668-9865152cf25d-istio-podinfo" pod "9f71d479-bb83-40b8-a668-9865152cf25d" (UID: "9f71d479-bb83-40b8-a668-9865152cf25d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:31.804718 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.804679    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-certs" pod "9f71d479-bb83-40b8-a668-9865152cf25d" (UID: "9f71d479-bb83-40b8-a668-9865152cf25d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.820525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.820281    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-istio-token" pod "9f71d479-bb83-40b8-a668-9865152cf25d" (UID: "9f71d479-bb83-40b8-a668-9865152cf25d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:31.820936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.820891    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-kube-api-access-g94kw" pod "17041e62-9718-42d3-9481-30851a9c7db0" (UID: "17041e62-9718-42d3-9481-30851a9c7db0"). InnerVolumeSpecName "kube-api-access-g94kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:31.820936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.820912    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-istio-token" pod "17041e62-9718-42d3-9481-30851a9c7db0" (UID: "17041e62-9718-42d3-9481-30851a9c7db0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:31.823184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.822566    5787 generic.go:356] "Generic (PLEG): container finished" podID="9f71d479-bb83-40b8-a668-9865152cf25d" containerID="c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6" exitCode=0
Oct 31 13:16:31.823184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.822637    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"
Oct 31 13:16:31.823184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.822660    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" event={"ID":"9f71d479-bb83-40b8-a668-9865152cf25d","Type":"ContainerDied","Data":"c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6"}
Oct 31 13:16:31.823184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.822904    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t" event={"ID":"9f71d479-bb83-40b8-a668-9865152cf25d","Type":"ContainerDied","Data":"c0fb77113c9475b87fc2f15c61bf5997ab2249f832c672b002d28a5c8f4dfc25"}
Oct 31 13:16:31.823184 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.822942    5787 scope.go:122] "RemoveContainer" containerID="c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6"
Oct 31 13:16:31.828024 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.827734    5787 generic.go:356] "Generic (PLEG): container finished" podID="17041e62-9718-42d3-9481-30851a9c7db0" containerID="932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37" exitCode=0
Oct 31 13:16:31.828024 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.827842    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17041e62-9718-42d3-9481-30851a9c7db0-istio-podinfo" pod "17041e62-9718-42d3-9481-30851a9c7db0" (UID: "17041e62-9718-42d3-9481-30851a9c7db0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 13:16:31.828024 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.827932    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" event={"ID":"17041e62-9718-42d3-9481-30851a9c7db0","Type":"ContainerDied","Data":"932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37"}
Oct 31 13:16:31.828024 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.827981    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5" event={"ID":"17041e62-9718-42d3-9481-30851a9c7db0","Type":"ContainerDied","Data":"8f17848852bee00e4aff29b01605ff843c29a5d9c29c2bdcf898ca1cd7557f88"}
Oct 31 13:16:31.828292 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.828062    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"
Oct 31 13:16:31.831998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.831309    5787 generic.go:356] "Generic (PLEG): container finished" podID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerID="82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b" exitCode=0
Oct 31 13:16:31.831998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.831452    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" event={"ID":"bc4b5f7e-1bed-46dd-b27f-e89f250107ec","Type":"ContainerDied","Data":"82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b"}
Oct 31 13:16:31.831998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.831499    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz" event={"ID":"bc4b5f7e-1bed-46dd-b27f-e89f250107ec","Type":"ContainerDied","Data":"5ce283cff5ed32fb71954bd077319bfeca4c71ea5c733453145f31dc1b616ba3"}
Oct 31 13:16:31.831998 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.831679    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"
Oct 31 13:16:31.850021 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.849597    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-envoy" pod "9f71d479-bb83-40b8-a668-9865152cf25d" (UID: "9f71d479-bb83-40b8-a668-9865152cf25d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:16:31.869053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.869018    5787 scope.go:122] "RemoveContainer" containerID="c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6"
Oct 31 13:16:31.869628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.869598    5787 generic.go:356] "Generic (PLEG): container finished" podID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerID="3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db" exitCode=0
Oct 31 13:16:31.877584 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:31.870371    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6\": container with ID starting with c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6 not found: ID does not exist" containerID="c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6"
Oct 31 13:16:31.877584 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.877546    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6"} err="failed to get container status \"c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6\": rpc error: code = NotFound desc = could not find container \"c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6\": container with ID starting with c46808d824e86ae56c708a040eaac89b02f5679ac79b3c794b0e29d79ebc37a6 not found: ID does not exist"
Oct 31 13:16:31.877842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.877586    5787 scope.go:122] "RemoveContainer" containerID="932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37"
Oct 31 13:16:31.877842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.870463    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" event={"ID":"b687cd60-1308-468e-a24f-674c6cef2f8a","Type":"ContainerDied","Data":"3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db"}
Oct 31 13:16:31.877842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.877765    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck" event={"ID":"b687cd60-1308-468e-a24f-674c6cef2f8a","Type":"ContainerDied","Data":"cf9cf04566f9d9bce273dc89f5ed16b572ff3f8a6dd45e15462d9b4614bc14cc"}
Oct 31 13:16:31.877842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.870577    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"
Oct 31 13:16:31.879610 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.879582    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.882234 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.882201    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f71d479-bb83-40b8-a668-9865152cf25d-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.885613 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.885569    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.886687 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.886665    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.886888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.886855    5787 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-data\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887020 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887005    5787 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887120 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887107    5787 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17041e62-9718-42d3-9481-30851a9c7db0-istiod-ca-cert\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887208 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887195    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887298 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887286    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887386 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887371    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g94kw\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-kube-api-access-g94kw\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887469 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887457    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f71d479-bb83-40b8-a668-9865152cf25d-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887548    5787 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17041e62-9718-42d3-9481-30851a9c7db0-istio-token\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887654 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887642    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887754    5787 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-workload-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887854    5787 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17041e62-9718-42d3-9481-30851a9c7db0-istio-podinfo\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.887954 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.887943    5787 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17041e62-9718-42d3-9481-30851a9c7db0-istio-envoy\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.888044 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.888033    5787 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f71d479-bb83-40b8-a668-9865152cf25d-credential-socket\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.888930 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.888193    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4662\" (UniqueName: \"kubernetes.io/projected/9f71d479-bb83-40b8-a668-9865152cf25d-kube-api-access-q4662\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:31.911490 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.911458    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"]
Oct 31 13:16:31.913236 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.913218    5787 scope.go:122] "RemoveContainer" containerID="932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37"
Oct 31 13:16:31.913671 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:31.913650    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37\": container with ID starting with 932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37 not found: ID does not exist" containerID="932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37"
Oct 31 13:16:31.913838 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.913813    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37"} err="failed to get container status \"932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37\": rpc error: code = NotFound desc = could not find container \"932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37\": container with ID starting with 932802bbb41152624dd8f28d4df5752eae62808c518e03bfc249fad20a6cdd37 not found: ID does not exist"
Oct 31 13:16:31.913928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.913918    5787 scope.go:122] "RemoveContainer" containerID="82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b"
Oct 31 13:16:31.917297 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.917262    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mrjh5"]
Oct 31 13:16:31.929236 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.929215    5787 scope.go:122] "RemoveContainer" containerID="82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b"
Oct 31 13:16:31.930154 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:31.930128    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b\": container with ID starting with 82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b not found: ID does not exist" containerID="82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b"
Oct 31 13:16:31.930337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.930310    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b"} err="failed to get container status \"82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b\": rpc error: code = NotFound desc = could not find container \"82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b\": container with ID starting with 82f609630e08fb5d0f58ef1228751adb9d0644b7d92f087645cb8a89524a0a7b not found: ID does not exist"
Oct 31 13:16:31.930455 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.930444    5787 scope.go:122] "RemoveContainer" containerID="3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db"
Oct 31 13:16:31.930609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.930247    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"]
Oct 31 13:16:31.938743 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.935361    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-rngkz"]
Oct 31 13:16:31.958794 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.958759    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"]
Oct 31 13:16:31.972339 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.972313    5787 scope.go:122] "RemoveContainer" containerID="3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db"
Oct 31 13:16:31.973025 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:31.972949    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db\": container with ID starting with 3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db not found: ID does not exist" containerID="3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db"
Oct 31 13:16:31.973025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.972988    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db"} err="failed to get container status \"3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db\": rpc error: code = NotFound desc = could not find container \"3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db\": container with ID starting with 3b9404c4e82bd54425059675404f460c6abd666035d6635b6334cb26c6a727db not found: ID does not exist"
Oct 31 13:16:31.977900 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:31.976559    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-mcpck"]
Oct 31 13:16:32.175323 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:32.174755    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"]
Oct 31 13:16:32.199829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:32.199776    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7bf66c-openshift-default-b6bbf7965-4db8t"]
Oct 31 13:16:32.716020 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:32.715986    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17041e62-9718-42d3-9481-30851a9c7db0" path="/var/lib/kubelet/pods/17041e62-9718-42d3-9481-30851a9c7db0/volumes"
Oct 31 13:16:32.716453 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:32.716436    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" path="/var/lib/kubelet/pods/9f71d479-bb83-40b8-a668-9865152cf25d/volumes"
Oct 31 13:16:32.716869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:32.716854    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" path="/var/lib/kubelet/pods/b687cd60-1308-468e-a24f-674c6cef2f8a/volumes"
Oct 31 13:16:32.717556 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:32.717537    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" path="/var/lib/kubelet/pods/bc4b5f7e-1bed-46dd-b27f-e89f250107ec/volumes"
Oct 31 13:16:48.246834 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:16:48.246789    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:16:50.999639 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:50.999599    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"]
Oct 31 13:16:51.000061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000033    5787 cpu_manager.go:414] "containerMap: removing container" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main"
Oct 31 13:16:51.000061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000045    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main"
Oct 31 13:16:51.000061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000059    5787 cpu_manager.go:414] "containerMap: removing container" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000064    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000073    5787 cpu_manager.go:414] "containerMap: removing container" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000078    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000085    5787 cpu_manager.go:414] "containerMap: removing container" podUID="a6893107-8ce0-40a6-846e-a7eecfdbe6f5" containerName="payload-processing"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000090    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6893107-8ce0-40a6-846e-a7eecfdbe6f5" containerName="payload-processing"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000098    5787 cpu_manager.go:414] "containerMap: removing container" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000102    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000109    5787 cpu_manager.go:414] "containerMap: removing container" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000114    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000120    5787 cpu_manager.go:414] "containerMap: removing container" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000124    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000133    5787 cpu_manager.go:414] "containerMap: removing container" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000136    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000143    5787 cpu_manager.go:414] "containerMap: removing container" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000147    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000153    5787 cpu_manager.go:414] "containerMap: removing container" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000157    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000165    5787 cpu_manager.go:414] "containerMap: removing container" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000169    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000174    5787 cpu_manager.go:414] "containerMap: removing container" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000178    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000188    5787 cpu_manager.go:414] "containerMap: removing container" podUID="3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa" containerName="revoke-keys"
Oct 31 13:16:51.000182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000192    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa" containerName="revoke-keys"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000198    5787 cpu_manager.go:414] "containerMap: removing container" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000202    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000214    5787 cpu_manager.go:414] "containerMap: removing container" podUID="b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" containerName="payload-pre-processing"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000218    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" containerName="payload-pre-processing"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000224    5787 cpu_manager.go:414] "containerMap: removing container" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000229    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000236    5787 cpu_manager.go:414] "containerMap: removing container" podUID="cfa8967d-a072-4b5a-8322-0769b0df3578" containerName="revoke-keys"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000240    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa8967d-a072-4b5a-8322-0769b0df3578" containerName="revoke-keys"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000248    5787 cpu_manager.go:414] "containerMap: removing container" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000252    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000258    5787 cpu_manager.go:414] "containerMap: removing container" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000262    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000269    5787 cpu_manager.go:414] "containerMap: removing container" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000273    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000282    5787 cpu_manager.go:414] "containerMap: removing container" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000286    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000293    5787 cpu_manager.go:414] "containerMap: removing container" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000297    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000304    5787 cpu_manager.go:414] "containerMap: removing container" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000308    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000315    5787 cpu_manager.go:414] "containerMap: removing container" podUID="b56bc1f6-53e8-4962-b690-46ed63a7ee4d" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000319    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56bc1f6-53e8-4962-b690-46ed63a7ee4d" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000324    5787 cpu_manager.go:414] "containerMap: removing container" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000328    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000334    5787 cpu_manager.go:414] "containerMap: removing container" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000339    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000345    5787 cpu_manager.go:414] "containerMap: removing container" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000349    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000356    5787 cpu_manager.go:414] "containerMap: removing container" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000360    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000367    5787 cpu_manager.go:414] "containerMap: removing container" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000371    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000425    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e8209df-eac1-4874-a784-fe7e9a33a7fa" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000436    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000441    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f71d479-bb83-40b8-a668-9865152cf25d" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000448    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa" containerName="revoke-keys"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000453    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="b889c3d9-985e-45d9-941e-539936378181" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000458    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7cda9f0-fd0c-4fc0-9c27-3435ea9dec75" containerName="payload-pre-processing"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000463    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c308e8e-d08e-43d5-accf-e9e9baa052b4" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000468    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa8967d-a072-4b5a-8322-0769b0df3578" containerName="revoke-keys"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000473    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a1da508-b47c-4513-9584-a5b4771d4f5d" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000478    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="527f4c39-43e6-4506-b95d-42423d24c688" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000483    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000488    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="144afe47-505e-436b-a07d-5e1ea36818a1" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000494    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="17041e62-9718-42d3-9481-30851a9c7db0" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000499    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb5a93b-7e4d-4eab-a4ea-fceaa2a5186b" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000504    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56bc1f6-53e8-4962-b690-46ed63a7ee4d" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000509    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="b687cd60-1308-468e-a24f-674c6cef2f8a" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000515    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="27fe87e5-d534-4f37-8260-5d5b415df0ff" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000521    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="365f64c7-a490-4ee0-87e1-56c68d22d5f3" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000525    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd55913-612c-4bc3-a067-d11002f9f182" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000530    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="debe71f3-91f0-48ac-b79b-85472ff7defd" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000535    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="23a61fd1-6bc0-4ea5-a1ad-a5cc957a76b0" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000540    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc4b5f7e-1bed-46dd-b27f-e89f250107ec" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000544    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6893107-8ce0-40a6-846e-a7eecfdbe6f5" containerName="payload-processing"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000549    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e7efa4-f7a0-4268-8356-91ef1caaf034" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000554    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9a093a-89b7-48ae-9123-d5b87aa9dd27" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000558    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="9206bfbe-392b-4a0e-972e-618cbafa13f9" containerName="istio-proxy"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000564    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd95bfe-b694-41a9-bf39-ad654cedee39" containerName="main"
Oct 31 13:16:51.000882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.000568    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f8f023-3eb8-429c-8a06-d9a39fe51981" containerName="istio-proxy"
Oct 31 13:16:51.004085 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.004066    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.010027 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.009986    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:16:51.013213 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.013186    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:16:51.013348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.013237    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:16:51.013348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.013237    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:16:51.014222 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.014199    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"]
Oct 31 13:16:51.105050 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.105012    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-tmczb\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.105205 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.105153    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d6lv\" (UniqueName: \"kubernetes.io/projected/64bf472f-5874-4a45-806c-49bd62934c1e-kube-api-access-7d6lv\") pod \"maas-api-revoke-sub-85237e45-tmczb\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.105205 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.105186    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-tmczb\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.206571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.206532    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d6lv\" (UniqueName: \"kubernetes.io/projected/64bf472f-5874-4a45-806c-49bd62934c1e-kube-api-access-7d6lv\") pod \"maas-api-revoke-sub-85237e45-tmczb\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.206571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.206574    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-tmczb\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.206833 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.206629    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-tmczb\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.207429 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.207408    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-tmczb\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.209239 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.209220    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-tmczb\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.221276 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.221246    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d6lv\" (UniqueName: \"kubernetes.io/projected/64bf472f-5874-4a45-806c-49bd62934c1e-kube-api-access-7d6lv\") pod \"maas-api-revoke-sub-85237e45-tmczb\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.320076 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.319990    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:51.601014 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:16:51.600978    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64bf472f_5874_4a45_806c_49bd62934c1e.slice/crio-0d1ca649b30041936762dfa093d0407e33fe897c0e0a4c1c8a38aaa417ee04af WatchSource:0}: Error finding container 0d1ca649b30041936762dfa093d0407e33fe897c0e0a4c1c8a38aaa417ee04af: Status 404 returned error can't find the container with id 0d1ca649b30041936762dfa093d0407e33fe897c0e0a4c1c8a38aaa417ee04af
Oct 31 13:16:51.601362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:51.601340    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"]
Oct 31 13:16:52.049569 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:52.049527    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb" event={"ID":"64bf472f-5874-4a45-806c-49bd62934c1e","Type":"ContainerStarted","Data":"cfe4e242df9877f1578416f9869151a544eee9b7f114664c329d843e28b7f2d8"}
Oct 31 13:16:52.050020 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:52.049596    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb" event={"ID":"64bf472f-5874-4a45-806c-49bd62934c1e","Type":"ContainerStarted","Data":"0d1ca649b30041936762dfa093d0407e33fe897c0e0a4c1c8a38aaa417ee04af"}
Oct 31 13:16:52.070517 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:52.070462    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb" podStartSLOduration=2.070443805 podStartE2EDuration="2.070443805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:16:52.067650593 +0000 UTC m=+991.483592434" watchObservedRunningTime="2026-10-31 13:16:52.070443805 +0000 UTC m=+991.486385644"
Oct 31 13:16:53.061254 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:53.061218    5787 generic.go:356] "Generic (PLEG): container finished" podID="64bf472f-5874-4a45-806c-49bd62934c1e" containerID="cfe4e242df9877f1578416f9869151a544eee9b7f114664c329d843e28b7f2d8" exitCode=0
Oct 31 13:16:53.061795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:53.061307    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb" event={"ID":"64bf472f-5874-4a45-806c-49bd62934c1e","Type":"ContainerDied","Data":"cfe4e242df9877f1578416f9869151a544eee9b7f114664c329d843e28b7f2d8"}
Oct 31 13:16:54.265840 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.265808    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:54.440120 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.440059    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-xks-ca\") pod \"64bf472f-5874-4a45-806c-49bd62934c1e\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") "
Oct 31 13:16:54.440347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.440194    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-openshift-ca\") pod \"64bf472f-5874-4a45-806c-49bd62934c1e\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") "
Oct 31 13:16:54.440347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.440261    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64bf472f-5874-4a45-806c-49bd62934c1e-kube-api-access-7d6lv\" (UniqueName: \"kubernetes.io/projected/64bf472f-5874-4a45-806c-49bd62934c1e-kube-api-access-7d6lv\") pod \"64bf472f-5874-4a45-806c-49bd62934c1e\" (UID: \"64bf472f-5874-4a45-806c-49bd62934c1e\") "
Oct 31 13:16:54.441000 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.440951    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-openshift-ca" pod "64bf472f-5874-4a45-806c-49bd62934c1e" (UID: "64bf472f-5874-4a45-806c-49bd62934c1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:16:54.443653 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.443622    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64bf472f-5874-4a45-806c-49bd62934c1e-kube-api-access-7d6lv" pod "64bf472f-5874-4a45-806c-49bd62934c1e" (UID: "64bf472f-5874-4a45-806c-49bd62934c1e"). InnerVolumeSpecName "kube-api-access-7d6lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:16:54.444079 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.444041    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-xks-ca" pod "64bf472f-5874-4a45-806c-49bd62934c1e" (UID: "64bf472f-5874-4a45-806c-49bd62934c1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:16:54.541330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.541227    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:54.541330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.541271    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64bf472f-5874-4a45-806c-49bd62934c1e-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:54.541330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:54.541288    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d6lv\" (UniqueName: \"kubernetes.io/projected/64bf472f-5874-4a45-806c-49bd62934c1e-kube-api-access-7d6lv\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:16:55.087399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:55.087276    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"
Oct 31 13:16:55.087586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:55.087307    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb" event={"ID":"64bf472f-5874-4a45-806c-49bd62934c1e","Type":"ContainerDied","Data":"0d1ca649b30041936762dfa093d0407e33fe897c0e0a4c1c8a38aaa417ee04af"}
Oct 31 13:16:55.087586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:16:55.087458    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d1ca649b30041936762dfa093d0407e33fe897c0e0a4c1c8a38aaa417ee04af"
Oct 31 13:17:16.038602 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:16.038514    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"]
Oct 31 13:17:16.045480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:16.045449    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"]
Oct 31 13:17:16.049856 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:16.049824    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5ac55d3-vw2zd"]
Oct 31 13:17:16.053098 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:16.053078    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a1fdcc8-nwd8v"]
Oct 31 13:17:16.704019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:16.703986    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1804f562-6f27-4a45-93ce-887da37f1590" path="/var/lib/kubelet/pods/1804f562-6f27-4a45-93ce-887da37f1590/volumes"
Oct 31 13:17:16.704397 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:16.704384    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb10808-c52a-4e25-b152-50f946432b19" path="/var/lib/kubelet/pods/efb10808-c52a-4e25-b152-50f946432b19/volumes"
Oct 31 13:17:18.246783 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:17:18.246745    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:17:21.027119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:21.027039    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"]
Oct 31 13:17:21.030322 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:21.030301    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24bd05d4-jmjfh"]
Oct 31 13:17:22.704538 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:22.704501    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ac0f77-88ae-4123-9418-82502b4eb3bf" path="/var/lib/kubelet/pods/53ac0f77-88ae-4123-9418-82502b4eb3bf/volumes"
Oct 31 13:17:25.025591 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:25.025547    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"]
Oct 31 13:17:25.030696 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:25.030667    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55d3f12d-fclbf"]
Oct 31 13:17:26.704617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:26.704574    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83df7efa-30cc-4341-825e-934fa3fc5eeb" path="/var/lib/kubelet/pods/83df7efa-30cc-4341-825e-934fa3fc5eeb/volumes"
Oct 31 13:17:30.025800 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:30.025759    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"]
Oct 31 13:17:30.028778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:30.028740    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6a0499-lzmm5"]
Oct 31 13:17:30.705273 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:30.705239    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37892070-9deb-4766-aa50-232e30892617" path="/var/lib/kubelet/pods/37892070-9deb-4766-aa50-232e30892617/volumes"
Oct 31 13:17:31.026564 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:31.026476    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"]
Oct 31 13:17:31.029842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:31.029816    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a578352e-m554j"]
Oct 31 13:17:32.027921 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:32.027879    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"]
Oct 31 13:17:32.033254 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:32.033219    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"]
Oct 31 13:17:32.036698 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:32.036669    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-92a717fc-c64tj"]
Oct 31 13:17:32.039656 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:32.039635    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09c1b14b-njqmc"]
Oct 31 13:17:32.704973 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:32.704939    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b71e93b-d7fc-4158-aef8-8e1cf1b6138c" path="/var/lib/kubelet/pods/5b71e93b-d7fc-4158-aef8-8e1cf1b6138c/volumes"
Oct 31 13:17:32.705372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:32.705359    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96c93acf-213a-4eaf-8cec-d9f895ea34aa" path="/var/lib/kubelet/pods/96c93acf-213a-4eaf-8cec-d9f895ea34aa/volumes"
Oct 31 13:17:32.705746 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:32.705733    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a47f6dd5-cd11-4f17-bce4-1497bb5b9769" path="/var/lib/kubelet/pods/a47f6dd5-cd11-4f17-bce4-1497bb5b9769/volumes"
Oct 31 13:17:37.027439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:37.027402    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"]
Oct 31 13:17:37.031081 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:37.031047    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12700de6-69qm5"]
Oct 31 13:17:38.029725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:38.029658    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"]
Oct 31 13:17:38.034156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:38.034122    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ae3e4b5-xgrpx"]
Oct 31 13:17:38.714179 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:38.714136    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d6c6bf-6c92-4e1c-8271-10f521baa944" path="/var/lib/kubelet/pods/43d6c6bf-6c92-4e1c-8271-10f521baa944/volumes"
Oct 31 13:17:38.714821 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:38.714799    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dd9a17d-ddbf-426d-bc71-df1139f8823e" path="/var/lib/kubelet/pods/6dd9a17d-ddbf-426d-bc71-df1139f8823e/volumes"
Oct 31 13:17:44.312374 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.312336    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"]
Oct 31 13:17:44.312810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.312765    5787 cpu_manager.go:414] "containerMap: removing container" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy"
Oct 31 13:17:44.312810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.312776    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy"
Oct 31 13:17:44.312810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.312791    5787 cpu_manager.go:414] "containerMap: removing container" podUID="64bf472f-5874-4a45-806c-49bd62934c1e" containerName="revoke-keys"
Oct 31 13:17:44.312810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.312796    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64bf472f-5874-4a45-806c-49bd62934c1e" containerName="revoke-keys"
Oct 31 13:17:44.312971 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.312892    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="64bf472f-5874-4a45-806c-49bd62934c1e" containerName="revoke-keys"
Oct 31 13:17:44.312971 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.312904    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="cca15df2-3a97-4142-8a38-8a0053421cf7" containerName="istio-proxy"
Oct 31 13:17:44.312971 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.312916    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d9a104-017d-48ce-8831-5d9e0635b637" containerName="istio-proxy"
Oct 31 13:17:44.316544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.316526    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.322780 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.322753    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:17:44.322968 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.322838    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:17:44.326268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.326237    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"]
Oct 31 13:17:44.352217 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.352182    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tllhp\" (UniqueName: \"kubernetes.io/projected/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-kube-api-access-tllhp\") pod \"maas-api-revoke-sub-c1e271a9-gl8l6\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.352399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.352308    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gl8l6\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.352399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.352342    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gl8l6\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.412116 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.412087    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:17:44.412421 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.412403    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:17:44.453889 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.453850    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gl8l6\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.454088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.453893    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gl8l6\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.454088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.453953    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tllhp\" (UniqueName: \"kubernetes.io/projected/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-kube-api-access-tllhp\") pod \"maas-api-revoke-sub-c1e271a9-gl8l6\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.454615 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.454594    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-gl8l6\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.456362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.456335    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-gl8l6\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.468141 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.468112    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tllhp\" (UniqueName: \"kubernetes.io/projected/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-kube-api-access-tllhp\") pod \"maas-api-revoke-sub-c1e271a9-gl8l6\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.631931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.631834    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:44.914386 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:44.914353    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"]
Oct 31 13:17:44.915154 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:17:44.915127    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce4f8b7c_6cf4_44f0_aba7_9f48d798d425.slice/crio-d96598d52998ef547089c95af2f4d91a9bdecc9a7519b41f8c67a1737d641db2 WatchSource:0}: Error finding container d96598d52998ef547089c95af2f4d91a9bdecc9a7519b41f8c67a1737d641db2: Status 404 returned error can't find the container with id d96598d52998ef547089c95af2f4d91a9bdecc9a7519b41f8c67a1737d641db2
Oct 31 13:17:45.026999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:45.026959    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"]
Oct 31 13:17:45.038114 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:45.038012    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"]
Oct 31 13:17:45.040553 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:45.040526    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-549f0286-gs7bd"]
Oct 31 13:17:45.043238 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:45.043212    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98b1ea3b-sr9dg"]
Oct 31 13:17:45.448498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:45.448451    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6" event={"ID":"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425","Type":"ContainerStarted","Data":"15950c801495e20c0dc95d0bd080462b430892128f88c823ef76840dbb29394e"}
Oct 31 13:17:45.448911 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:45.448551    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6" event={"ID":"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425","Type":"ContainerStarted","Data":"d96598d52998ef547089c95af2f4d91a9bdecc9a7519b41f8c67a1737d641db2"}
Oct 31 13:17:45.470245 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:45.470187    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6" podStartSLOduration=1.470169397 podStartE2EDuration="1.470169397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:17:45.467966541 +0000 UTC m=+1044.883908380" watchObservedRunningTime="2026-10-31 13:17:45.470169397 +0000 UTC m=+1044.886111276"
Oct 31 13:17:46.028120 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:46.028082    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"]
Oct 31 13:17:46.033526 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:46.033497    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71bc3bd7-r5w56"]
Oct 31 13:17:46.456465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:46.456425    5787 generic.go:356] "Generic (PLEG): container finished" podID="ce4f8b7c-6cf4-44f0-aba7-9f48d798d425" containerID="15950c801495e20c0dc95d0bd080462b430892128f88c823ef76840dbb29394e" exitCode=0
Oct 31 13:17:46.456940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:46.456515    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6" event={"ID":"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425","Type":"ContainerDied","Data":"15950c801495e20c0dc95d0bd080462b430892128f88c823ef76840dbb29394e"}
Oct 31 13:17:46.704496 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:46.704463    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0645129c-18a5-43c3-9227-dc509a9677a9" path="/var/lib/kubelet/pods/0645129c-18a5-43c3-9227-dc509a9677a9/volumes"
Oct 31 13:17:46.704901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:46.704887    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4b605c1-f2f0-4dd9-b26c-32dece01d557" path="/var/lib/kubelet/pods/a4b605c1-f2f0-4dd9-b26c-32dece01d557/volumes"
Oct 31 13:17:46.705238 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:46.705226    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d88165-8e3b-4486-9f0b-2ac58cb80b29" path="/var/lib/kubelet/pods/c1d88165-8e3b-4486-9f0b-2ac58cb80b29/volumes"
Oct 31 13:17:47.607141 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.607111    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:47.686302 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.686269    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-kube-api-access-tllhp\" (UniqueName: \"kubernetes.io/projected/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-kube-api-access-tllhp\") pod \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") "
Oct 31 13:17:47.686492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.686421    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-openshift-ca\") pod \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") "
Oct 31 13:17:47.686492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.686446    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-xks-ca\") pod \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\" (UID: \"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425\") "
Oct 31 13:17:47.686857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.686836    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-openshift-ca" pod "ce4f8b7c-6cf4-44f0-aba7-9f48d798d425" (UID: "ce4f8b7c-6cf4-44f0-aba7-9f48d798d425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:17:47.688645 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.688626    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-kube-api-access-tllhp" pod "ce4f8b7c-6cf4-44f0-aba7-9f48d798d425" (UID: "ce4f8b7c-6cf4-44f0-aba7-9f48d798d425"). InnerVolumeSpecName "kube-api-access-tllhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:17:47.688756 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.688734    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-xks-ca" pod "ce4f8b7c-6cf4-44f0-aba7-9f48d798d425" (UID: "ce4f8b7c-6cf4-44f0-aba7-9f48d798d425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:17:47.787699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.787614    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:17:47.787699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.787646    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:17:47.787699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:47.787656    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tllhp\" (UniqueName: \"kubernetes.io/projected/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425-kube-api-access-tllhp\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:17:48.029972 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:48.029932    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"]
Oct 31 13:17:48.032623 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:48.032599    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f31cd2ce-dbfvd"]
Oct 31 13:17:48.245935 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:17:48.245892    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:17:48.471699 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:48.471666    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"
Oct 31 13:17:48.471912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:48.471699    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6" event={"ID":"ce4f8b7c-6cf4-44f0-aba7-9f48d798d425","Type":"ContainerDied","Data":"d96598d52998ef547089c95af2f4d91a9bdecc9a7519b41f8c67a1737d641db2"}
Oct 31 13:17:48.471912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:48.471782    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d96598d52998ef547089c95af2f4d91a9bdecc9a7519b41f8c67a1737d641db2"
Oct 31 13:17:48.704300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:48.704270    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d670a58-5e6a-4e27-af41-921f20b393a2" path="/var/lib/kubelet/pods/6d670a58-5e6a-4e27-af41-921f20b393a2/volumes"
Oct 31 13:17:50.032132 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:50.032086    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"]
Oct 31 13:17:50.034833 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:50.034808    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c95abd5-4rkzx"]
Oct 31 13:17:50.704531 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:50.704499    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18475b18-2d0a-4946-926a-812bdd8ddce0" path="/var/lib/kubelet/pods/18475b18-2d0a-4946-926a-812bdd8ddce0/volumes"
Oct 31 13:17:54.026773 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:54.026735    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"]
Oct 31 13:17:54.029369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:54.029345    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3192b15-m6vgc"]
Oct 31 13:17:54.705026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:17:54.704989    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f4efea5-a9c8-4060-a56c-717040239eac" path="/var/lib/kubelet/pods/6f4efea5-a9c8-4060-a56c-717040239eac/volumes"
Oct 31 13:18:01.028047 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:01.028004    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"]
Oct 31 13:18:01.031755 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:01.031727    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69846f89-47nlm"]
Oct 31 13:18:02.704432 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:02.704399    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="525ea675-86c9-4900-b9c1-67bc9a203896" path="/var/lib/kubelet/pods/525ea675-86c9-4900-b9c1-67bc9a203896/volumes"
Oct 31 13:18:04.027158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:04.027121    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"]
Oct 31 13:18:04.030369 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:04.030340    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-998e19c8-5gh7s"]
Oct 31 13:18:04.704611 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:04.704573    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7965ee70-9bad-459e-8709-ac3a1d3cbf8f" path="/var/lib/kubelet/pods/7965ee70-9bad-459e-8709-ac3a1d3cbf8f/volumes"
Oct 31 13:18:05.386678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.386636    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"]
Oct 31 13:18:05.387274 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.387253    5787 cpu_manager.go:414] "containerMap: removing container" podUID="ce4f8b7c-6cf4-44f0-aba7-9f48d798d425" containerName="revoke-keys"
Oct 31 13:18:05.387364 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.387296    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce4f8b7c-6cf4-44f0-aba7-9f48d798d425" containerName="revoke-keys"
Oct 31 13:18:05.387422 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.387412    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce4f8b7c-6cf4-44f0-aba7-9f48d798d425" containerName="revoke-keys"
Oct 31 13:18:05.390763 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.390740    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.397736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.397521    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:18:05.397736 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.397651    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:18:05.397920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.397840    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:18:05.398395 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.398014    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:18:05.402166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.402136    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"]
Oct 31 13:18:05.466248 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.466208    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswdg\" (UniqueName: \"kubernetes.io/projected/414013a4-02e7-4b4d-8596-966e5383cb53-kube-api-access-dswdg\") pod \"maas-api-revoke-sub-2e1f5b5f-wqgkw\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.466442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.466357    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wqgkw\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.466442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.466414    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wqgkw\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.567864 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.567824    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wqgkw\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.567864 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.567874    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dswdg\" (UniqueName: \"kubernetes.io/projected/414013a4-02e7-4b4d-8596-966e5383cb53-kube-api-access-dswdg\") pod \"maas-api-revoke-sub-2e1f5b5f-wqgkw\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.568110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.567949    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wqgkw\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.568595 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.568577    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wqgkw\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.570419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.570401    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-wqgkw\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.582089 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.582057    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswdg\" (UniqueName: \"kubernetes.io/projected/414013a4-02e7-4b4d-8596-966e5383cb53-kube-api-access-dswdg\") pod \"maas-api-revoke-sub-2e1f5b5f-wqgkw\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:05.706371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:05.706334    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:06.019223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:06.019191    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"]
Oct 31 13:18:06.020053 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:18:06.020029    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414013a4_02e7_4b4d_8596_966e5383cb53.slice/crio-574afb6210867ca72e227528022562a9919bbe9019ad5f716c48573ec81b018e WatchSource:0}: Error finding container 574afb6210867ca72e227528022562a9919bbe9019ad5f716c48573ec81b018e: Status 404 returned error can't find the container with id 574afb6210867ca72e227528022562a9919bbe9019ad5f716c48573ec81b018e
Oct 31 13:18:06.657053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:06.656958    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw" event={"ID":"414013a4-02e7-4b4d-8596-966e5383cb53","Type":"ContainerStarted","Data":"fcee3478b563ab83a655fa457ad6ca78562be641833411a45c37756e1a092994"}
Oct 31 13:18:06.657053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:06.657022    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw" event={"ID":"414013a4-02e7-4b4d-8596-966e5383cb53","Type":"ContainerStarted","Data":"574afb6210867ca72e227528022562a9919bbe9019ad5f716c48573ec81b018e"}
Oct 31 13:18:06.680966 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:06.680913    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw" podStartSLOduration=1.680900006 podStartE2EDuration="1.680900006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:06.676748464 +0000 UTC m=+1066.092690304" watchObservedRunningTime="2026-10-31 13:18:06.680900006 +0000 UTC m=+1066.096841881"
Oct 31 13:18:07.666088 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:07.666057    5787 generic.go:356] "Generic (PLEG): container finished" podID="414013a4-02e7-4b4d-8596-966e5383cb53" containerID="fcee3478b563ab83a655fa457ad6ca78562be641833411a45c37756e1a092994" exitCode=0
Oct 31 13:18:07.666502 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:07.666147    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw" event={"ID":"414013a4-02e7-4b4d-8596-966e5383cb53","Type":"ContainerDied","Data":"fcee3478b563ab83a655fa457ad6ca78562be641833411a45c37756e1a092994"}
Oct 31 13:18:08.810523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:08.810497    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:08.902550 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:08.902515    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/414013a4-02e7-4b4d-8596-966e5383cb53-kube-api-access-dswdg\" (UniqueName: \"kubernetes.io/projected/414013a4-02e7-4b4d-8596-966e5383cb53-kube-api-access-dswdg\") pod \"414013a4-02e7-4b4d-8596-966e5383cb53\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") "
Oct 31 13:18:08.902799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:08.902578    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-xks-ca\") pod \"414013a4-02e7-4b4d-8596-966e5383cb53\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") "
Oct 31 13:18:08.902799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:08.902725    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-openshift-ca\") pod \"414013a4-02e7-4b4d-8596-966e5383cb53\" (UID: \"414013a4-02e7-4b4d-8596-966e5383cb53\") "
Oct 31 13:18:08.903094 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:08.903069    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-openshift-ca" pod "414013a4-02e7-4b4d-8596-966e5383cb53" (UID: "414013a4-02e7-4b4d-8596-966e5383cb53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:18:08.904905 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:08.904883    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-xks-ca" pod "414013a4-02e7-4b4d-8596-966e5383cb53" (UID: "414013a4-02e7-4b4d-8596-966e5383cb53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:18:08.904967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:08.904954    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/414013a4-02e7-4b4d-8596-966e5383cb53-kube-api-access-dswdg" pod "414013a4-02e7-4b4d-8596-966e5383cb53" (UID: "414013a4-02e7-4b4d-8596-966e5383cb53"). InnerVolumeSpecName "kube-api-access-dswdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:18:09.003846 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:09.003803    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:09.003846 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:09.003834    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dswdg\" (UniqueName: \"kubernetes.io/projected/414013a4-02e7-4b4d-8596-966e5383cb53-kube-api-access-dswdg\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:09.003846 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:09.003847    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/414013a4-02e7-4b4d-8596-966e5383cb53-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:09.683525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:09.683495    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"
Oct 31 13:18:09.683525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:09.683509    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw" event={"ID":"414013a4-02e7-4b4d-8596-966e5383cb53","Type":"ContainerDied","Data":"574afb6210867ca72e227528022562a9919bbe9019ad5f716c48573ec81b018e"}
Oct 31 13:18:09.683784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:09.683552    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="574afb6210867ca72e227528022562a9919bbe9019ad5f716c48573ec81b018e"
Oct 31 13:18:14.033225 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:14.033189    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"]
Oct 31 13:18:14.035948 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:14.035918    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c4b8cef-sbxx7"]
Oct 31 13:18:14.704959 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:14.704916    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71ff155-9ce4-4f6b-892a-3a17dd7fe837" path="/var/lib/kubelet/pods/b71ff155-9ce4-4f6b-892a-3a17dd7fe837/volumes"
Oct 31 13:18:16.442677 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.442637    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"]
Oct 31 13:18:16.443129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.443075    5787 cpu_manager.go:414] "containerMap: removing container" podUID="414013a4-02e7-4b4d-8596-966e5383cb53" containerName="revoke-keys"
Oct 31 13:18:16.443129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.443088    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="414013a4-02e7-4b4d-8596-966e5383cb53" containerName="revoke-keys"
Oct 31 13:18:16.443218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.443178    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="414013a4-02e7-4b4d-8596-966e5383cb53" containerName="revoke-keys"
Oct 31 13:18:16.446450 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.446431    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.454820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.454795    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:18:16.455020 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.454817    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:18:16.455119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.454856    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:18:16.455119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.454950    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:18:16.458138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.458113    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"]
Oct 31 13:18:16.579246 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.579206    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5qtw\" (UniqueName: \"kubernetes.io/projected/6da40fb8-0a34-4f04-87bf-86e4d3559391-kube-api-access-m5qtw\") pod \"maas-api-revoke-sub-ec843018-hwxmh\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.579461 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.579254    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hwxmh\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.579461 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.579274    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hwxmh\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.680561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.680518    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hwxmh\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.680561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.680561    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hwxmh\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.680881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.680655    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5qtw\" (UniqueName: \"kubernetes.io/projected/6da40fb8-0a34-4f04-87bf-86e4d3559391-kube-api-access-m5qtw\") pod \"maas-api-revoke-sub-ec843018-hwxmh\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.681352 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.681331    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-hwxmh\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.683119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.683094    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-hwxmh\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.694801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.694737    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5qtw\" (UniqueName: \"kubernetes.io/projected/6da40fb8-0a34-4f04-87bf-86e4d3559391-kube-api-access-m5qtw\") pod \"maas-api-revoke-sub-ec843018-hwxmh\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:16.760991 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:16.760953    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:17.049263 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:17.049234    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"]
Oct 31 13:18:17.050901 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:18:17.050875    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da40fb8_0a34_4f04_87bf_86e4d3559391.slice/crio-da79075fdfeff7c881f4495eef3b6ddcc9f1a5330593344610363955bf447c48 WatchSource:0}: Error finding container da79075fdfeff7c881f4495eef3b6ddcc9f1a5330593344610363955bf447c48: Status 404 returned error can't find the container with id da79075fdfeff7c881f4495eef3b6ddcc9f1a5330593344610363955bf447c48
Oct 31 13:18:17.756911 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:17.756870    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh" event={"ID":"6da40fb8-0a34-4f04-87bf-86e4d3559391","Type":"ContainerStarted","Data":"5926d96fc945961b46f2b5a0508a4ea3b6a0cfb133521fb3f410c9174f8ddfcc"}
Oct 31 13:18:17.757301 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:17.756928    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh" event={"ID":"6da40fb8-0a34-4f04-87bf-86e4d3559391","Type":"ContainerStarted","Data":"da79075fdfeff7c881f4495eef3b6ddcc9f1a5330593344610363955bf447c48"}
Oct 31 13:18:17.777423 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:17.777371    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh" podStartSLOduration=1.777358005 podStartE2EDuration="1.777358005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:17.77599043 +0000 UTC m=+1077.191932284" watchObservedRunningTime="2026-10-31 13:18:17.777358005 +0000 UTC m=+1077.193299847"
Oct 31 13:18:18.245768 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:18:18.245730    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:18:18.768481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:18.768443    5787 generic.go:356] "Generic (PLEG): container finished" podID="6da40fb8-0a34-4f04-87bf-86e4d3559391" containerID="5926d96fc945961b46f2b5a0508a4ea3b6a0cfb133521fb3f410c9174f8ddfcc" exitCode=0
Oct 31 13:18:18.768931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:18.768525    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh" event={"ID":"6da40fb8-0a34-4f04-87bf-86e4d3559391","Type":"ContainerDied","Data":"5926d96fc945961b46f2b5a0508a4ea3b6a0cfb133521fb3f410c9174f8ddfcc"}
Oct 31 13:18:20.170881 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.170475    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:20.222248 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.222212    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-openshift-ca\") pod \"6da40fb8-0a34-4f04-87bf-86e4d3559391\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") "
Oct 31 13:18:20.222411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.222270    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da40fb8-0a34-4f04-87bf-86e4d3559391-kube-api-access-m5qtw\" (UniqueName: \"kubernetes.io/projected/6da40fb8-0a34-4f04-87bf-86e4d3559391-kube-api-access-m5qtw\") pod \"6da40fb8-0a34-4f04-87bf-86e4d3559391\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") "
Oct 31 13:18:20.222411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.222334    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-xks-ca\") pod \"6da40fb8-0a34-4f04-87bf-86e4d3559391\" (UID: \"6da40fb8-0a34-4f04-87bf-86e4d3559391\") "
Oct 31 13:18:20.222922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.222891    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-openshift-ca" pod "6da40fb8-0a34-4f04-87bf-86e4d3559391" (UID: "6da40fb8-0a34-4f04-87bf-86e4d3559391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:18:20.224777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.224753    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-xks-ca" pod "6da40fb8-0a34-4f04-87bf-86e4d3559391" (UID: "6da40fb8-0a34-4f04-87bf-86e4d3559391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:18:20.225348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.225324    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da40fb8-0a34-4f04-87bf-86e4d3559391-kube-api-access-m5qtw" pod "6da40fb8-0a34-4f04-87bf-86e4d3559391" (UID: "6da40fb8-0a34-4f04-87bf-86e4d3559391"). InnerVolumeSpecName "kube-api-access-m5qtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:18:20.323066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.323025    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:20.323066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.323057    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6da40fb8-0a34-4f04-87bf-86e4d3559391-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:20.323066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.323067    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5qtw\" (UniqueName: \"kubernetes.io/projected/6da40fb8-0a34-4f04-87bf-86e4d3559391-kube-api-access-m5qtw\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:20.829590 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.829559    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"
Oct 31 13:18:20.829904 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.829580    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh" event={"ID":"6da40fb8-0a34-4f04-87bf-86e4d3559391","Type":"ContainerDied","Data":"da79075fdfeff7c881f4495eef3b6ddcc9f1a5330593344610363955bf447c48"}
Oct 31 13:18:20.830021 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:20.829933    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da79075fdfeff7c881f4495eef3b6ddcc9f1a5330593344610363955bf447c48"
Oct 31 13:18:21.383883 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:21.383836    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-72d7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:18:21.384438 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:21.384074    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podUID="2b850a6e-358a-4baa-8730-9b569d126952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:18:21.959567 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:21.959533    5787 scope.go:122] "RemoveContainer" containerID="c4f3ac799b76290dab93e4a69f4211c02ae4562f7b66b2a074e3fa1fd8382c00"
Oct 31 13:18:21.969628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:21.969602    5787 scope.go:122] "RemoveContainer" containerID="f8d45d7e814ecc32ee0ae70dedea3d47c13668b48fd2eae5c22d6ff6b8c1f210"
Oct 31 13:18:21.979419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:21.979395    5787 scope.go:122] "RemoveContainer" containerID="27afeebd1dd5446f0c08a437493f8e8084ca0aa010ba4b54773d34fcdaa8e92a"
Oct 31 13:18:42.895161 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.895121    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"]
Oct 31 13:18:42.895739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.895627    5787 cpu_manager.go:414] "containerMap: removing container" podUID="6da40fb8-0a34-4f04-87bf-86e4d3559391" containerName="revoke-keys"
Oct 31 13:18:42.895739 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.895645    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da40fb8-0a34-4f04-87bf-86e4d3559391" containerName="revoke-keys"
Oct 31 13:18:42.895888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.895769    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da40fb8-0a34-4f04-87bf-86e4d3559391" containerName="revoke-keys"
Oct 31 13:18:42.899209 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.899185    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:42.907182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.906915    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:18:42.907182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.907090    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:18:42.907182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.907179    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:18:42.907399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.907225    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:18:42.909791 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:42.909767    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"]
Oct 31 13:18:43.072455 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.072410    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5ghr2\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.072638 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.072467    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh462\" (UniqueName: \"kubernetes.io/projected/3636f9e4-b2e7-466a-bddf-b756496fe3b9-kube-api-access-hh462\") pod \"maas-api-revoke-sub-046e1343-5ghr2\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.072638 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.072594    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5ghr2\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.174060 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.173975    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5ghr2\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.174226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.174055    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5ghr2\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.174226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.174100    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh462\" (UniqueName: \"kubernetes.io/projected/3636f9e4-b2e7-466a-bddf-b756496fe3b9-kube-api-access-hh462\") pod \"maas-api-revoke-sub-046e1343-5ghr2\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.174836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.174779    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5ghr2\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.176578 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.176550    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5ghr2\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.188302 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.188278    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh462\" (UniqueName: \"kubernetes.io/projected/3636f9e4-b2e7-466a-bddf-b756496fe3b9-kube-api-access-hh462\") pod \"maas-api-revoke-sub-046e1343-5ghr2\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.213258 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.213222    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:43.497836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.497806    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"]
Oct 31 13:18:43.498154 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:18:43.498131    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3636f9e4_b2e7_466a_bddf_b756496fe3b9.slice/crio-2a2f7cad5c586466dbbdef5b9652c180c1a8aa252623ccbd815d2ce233a360d1 WatchSource:0}: Error finding container 2a2f7cad5c586466dbbdef5b9652c180c1a8aa252623ccbd815d2ce233a360d1: Status 404 returned error can't find the container with id 2a2f7cad5c586466dbbdef5b9652c180c1a8aa252623ccbd815d2ce233a360d1
Oct 31 13:18:43.998163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.998063    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2" event={"ID":"3636f9e4-b2e7-466a-bddf-b756496fe3b9","Type":"ContainerStarted","Data":"3230c3beed5d9b1d888a2b2a647597a4dc88e8ec69349c91fc59d8290b09207f"}
Oct 31 13:18:43.998163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:43.998130    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2" event={"ID":"3636f9e4-b2e7-466a-bddf-b756496fe3b9","Type":"ContainerStarted","Data":"2a2f7cad5c586466dbbdef5b9652c180c1a8aa252623ccbd815d2ce233a360d1"}
Oct 31 13:18:44.020820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:44.020765    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2" podStartSLOduration=2.020751155 podStartE2EDuration="2.020751155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:44.018510708 +0000 UTC m=+1103.434452545" watchObservedRunningTime="2026-10-31 13:18:44.020751155 +0000 UTC m=+1103.436692995"
Oct 31 13:18:45.006042 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:45.006000    5787 generic.go:356] "Generic (PLEG): container finished" podID="3636f9e4-b2e7-466a-bddf-b756496fe3b9" containerID="3230c3beed5d9b1d888a2b2a647597a4dc88e8ec69349c91fc59d8290b09207f" exitCode=0
Oct 31 13:18:45.006445 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:45.006093    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2" event={"ID":"3636f9e4-b2e7-466a-bddf-b756496fe3b9","Type":"ContainerDied","Data":"3230c3beed5d9b1d888a2b2a647597a4dc88e8ec69349c91fc59d8290b09207f"}
Oct 31 13:18:46.146014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.145989    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:46.306606 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.306527    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-openshift-ca\") pod \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") "
Oct 31 13:18:46.306606 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.306567    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-xks-ca\") pod \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") "
Oct 31 13:18:46.306822 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.306676    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3636f9e4-b2e7-466a-bddf-b756496fe3b9-kube-api-access-hh462\" (UniqueName: \"kubernetes.io/projected/3636f9e4-b2e7-466a-bddf-b756496fe3b9-kube-api-access-hh462\") pod \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\" (UID: \"3636f9e4-b2e7-466a-bddf-b756496fe3b9\") "
Oct 31 13:18:46.306974 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.306950    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-openshift-ca" pod "3636f9e4-b2e7-466a-bddf-b756496fe3b9" (UID: "3636f9e4-b2e7-466a-bddf-b756496fe3b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:18:46.308916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.308892    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3636f9e4-b2e7-466a-bddf-b756496fe3b9-kube-api-access-hh462" pod "3636f9e4-b2e7-466a-bddf-b756496fe3b9" (UID: "3636f9e4-b2e7-466a-bddf-b756496fe3b9"). InnerVolumeSpecName "kube-api-access-hh462". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:18:46.308916 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.308907    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-xks-ca" pod "3636f9e4-b2e7-466a-bddf-b756496fe3b9" (UID: "3636f9e4-b2e7-466a-bddf-b756496fe3b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:18:46.407658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.407619    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:46.407658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.407650    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3636f9e4-b2e7-466a-bddf-b756496fe3b9-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:46.407658 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:46.407662    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh462\" (UniqueName: \"kubernetes.io/projected/3636f9e4-b2e7-466a-bddf-b756496fe3b9-kube-api-access-hh462\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:47.019532 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:47.019432    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2" event={"ID":"3636f9e4-b2e7-466a-bddf-b756496fe3b9","Type":"ContainerDied","Data":"2a2f7cad5c586466dbbdef5b9652c180c1a8aa252623ccbd815d2ce233a360d1"}
Oct 31 13:18:47.019532 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:47.019499    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a2f7cad5c586466dbbdef5b9652c180c1a8aa252623ccbd815d2ce233a360d1"
Oct 31 13:18:47.019532 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:47.019446    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"
Oct 31 13:18:48.245994 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:18:48.245957    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:18:53.130656 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.130621    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"]
Oct 31 13:18:53.141018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.140980    5787 cpu_manager.go:414] "containerMap: removing container" podUID="3636f9e4-b2e7-466a-bddf-b756496fe3b9" containerName="revoke-keys"
Oct 31 13:18:53.141018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.141011    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3636f9e4-b2e7-466a-bddf-b756496fe3b9" containerName="revoke-keys"
Oct 31 13:18:53.141259 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.141249    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="3636f9e4-b2e7-466a-bddf-b756496fe3b9" containerName="revoke-keys"
Oct 31 13:18:53.145910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.145882    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"]
Oct 31 13:18:53.146066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.145977    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.151601 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.151575    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:18:53.151601 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.151601    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:18:53.151922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.151629    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:18:53.151922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.151686    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:18:53.170920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.170889    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4qs\" (UniqueName: \"kubernetes.io/projected/63cb9375-5f78-4f12-906e-9f6e6b274a96-kube-api-access-6d4qs\") pod \"maas-api-revoke-sub-d3fba056-ttmjx\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.171150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.170938    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ttmjx\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.171150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.171000    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ttmjx\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.272025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.271981    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ttmjx\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.272238 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.272069    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4qs\" (UniqueName: \"kubernetes.io/projected/63cb9375-5f78-4f12-906e-9f6e6b274a96-kube-api-access-6d4qs\") pod \"maas-api-revoke-sub-d3fba056-ttmjx\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.272238 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.272119    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ttmjx\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.272771 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.272749    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-ttmjx\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.274641 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.274617    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-ttmjx\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.285603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.285571    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4qs\" (UniqueName: \"kubernetes.io/projected/63cb9375-5f78-4f12-906e-9f6e6b274a96-kube-api-access-6d4qs\") pod \"maas-api-revoke-sub-d3fba056-ttmjx\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.461957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.461927    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:53.748123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:53.748099    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"]
Oct 31 13:18:53.748980 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:18:53.748950    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63cb9375_5f78_4f12_906e_9f6e6b274a96.slice/crio-9e09de4d14f527bdf6c8c64a71bc133e445474af4fa02e72a5d61551c775f95a WatchSource:0}: Error finding container 9e09de4d14f527bdf6c8c64a71bc133e445474af4fa02e72a5d61551c775f95a: Status 404 returned error can't find the container with id 9e09de4d14f527bdf6c8c64a71bc133e445474af4fa02e72a5d61551c775f95a
Oct 31 13:18:54.068119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:54.068082    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx" event={"ID":"63cb9375-5f78-4f12-906e-9f6e6b274a96","Type":"ContainerStarted","Data":"f776201fcd19cbc137a0269320219eecc6ba90aa0c218bad3b0e684b8a78f215"}
Oct 31 13:18:54.068284 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:54.068139    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx" event={"ID":"63cb9375-5f78-4f12-906e-9f6e6b274a96","Type":"ContainerStarted","Data":"9e09de4d14f527bdf6c8c64a71bc133e445474af4fa02e72a5d61551c775f95a"}
Oct 31 13:18:54.090982 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:54.090934    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx" podStartSLOduration=1.090919901 podStartE2EDuration="1.090919901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:18:54.088579192 +0000 UTC m=+1113.504521030" watchObservedRunningTime="2026-10-31 13:18:54.090919901 +0000 UTC m=+1113.506861741"
Oct 31 13:18:55.079312 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:55.079271    5787 generic.go:356] "Generic (PLEG): container finished" podID="63cb9375-5f78-4f12-906e-9f6e6b274a96" containerID="f776201fcd19cbc137a0269320219eecc6ba90aa0c218bad3b0e684b8a78f215" exitCode=0
Oct 31 13:18:55.079883 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:55.079332    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx" event={"ID":"63cb9375-5f78-4f12-906e-9f6e6b274a96","Type":"ContainerDied","Data":"f776201fcd19cbc137a0269320219eecc6ba90aa0c218bad3b0e684b8a78f215"}
Oct 31 13:18:56.431782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.431757    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:56.517732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.517345    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-openshift-ca\") pod \"63cb9375-5f78-4f12-906e-9f6e6b274a96\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") "
Oct 31 13:18:56.517732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.517510    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-xks-ca\") pod \"63cb9375-5f78-4f12-906e-9f6e6b274a96\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") "
Oct 31 13:18:56.517732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.517562    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63cb9375-5f78-4f12-906e-9f6e6b274a96-kube-api-access-6d4qs\" (UniqueName: \"kubernetes.io/projected/63cb9375-5f78-4f12-906e-9f6e6b274a96-kube-api-access-6d4qs\") pod \"63cb9375-5f78-4f12-906e-9f6e6b274a96\" (UID: \"63cb9375-5f78-4f12-906e-9f6e6b274a96\") "
Oct 31 13:18:56.518399 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.518304    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-openshift-ca" pod "63cb9375-5f78-4f12-906e-9f6e6b274a96" (UID: "63cb9375-5f78-4f12-906e-9f6e6b274a96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:18:56.520531 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.520500    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63cb9375-5f78-4f12-906e-9f6e6b274a96-kube-api-access-6d4qs" pod "63cb9375-5f78-4f12-906e-9f6e6b274a96" (UID: "63cb9375-5f78-4f12-906e-9f6e6b274a96"). InnerVolumeSpecName "kube-api-access-6d4qs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:18:56.521023 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.521000    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-xks-ca" pod "63cb9375-5f78-4f12-906e-9f6e6b274a96" (UID: "63cb9375-5f78-4f12-906e-9f6e6b274a96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:18:56.618488 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.618449    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:56.618488 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.618486    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63cb9375-5f78-4f12-906e-9f6e6b274a96-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:56.618731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:56.618502    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d4qs\" (UniqueName: \"kubernetes.io/projected/63cb9375-5f78-4f12-906e-9f6e6b274a96-kube-api-access-6d4qs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:18:57.108153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:57.108068    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"
Oct 31 13:18:57.108153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:57.108096    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx" event={"ID":"63cb9375-5f78-4f12-906e-9f6e6b274a96","Type":"ContainerDied","Data":"9e09de4d14f527bdf6c8c64a71bc133e445474af4fa02e72a5d61551c775f95a"}
Oct 31 13:18:57.108153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:18:57.108146    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e09de4d14f527bdf6c8c64a71bc133e445474af4fa02e72a5d61551c775f95a"
Oct 31 13:19:06.029655 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:06.029621    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"]
Oct 31 13:19:06.033492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:06.033467    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b23d31bd-j77zp"]
Oct 31 13:19:06.706344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:06.706306    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a496c77d-33bb-4c88-8de7-1ffa82d0bdc4" path="/var/lib/kubelet/pods/a496c77d-33bb-4c88-8de7-1ffa82d0bdc4/volumes"
Oct 31 13:19:08.029154 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:08.029120    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"]
Oct 31 13:19:08.050746 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:08.050686    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef56e0e2-f728k"]
Oct 31 13:19:08.710866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:08.710824    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="841f919b-9011-4970-98aa-2d0cb8fe36f6" path="/var/lib/kubelet/pods/841f919b-9011-4970-98aa-2d0cb8fe36f6/volumes"
Oct 31 13:19:09.033489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:09.033403    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"]
Oct 31 13:19:09.037665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:09.037637    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aea6021-b24xb"]
Oct 31 13:19:10.704648 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:10.704617    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641a74ef-ffdc-417d-86be-b38c42d67de3" path="/var/lib/kubelet/pods/641a74ef-ffdc-417d-86be-b38c42d67de3/volumes"
Oct 31 13:19:13.947453 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.947416    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"]
Oct 31 13:19:13.947876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.947853    5787 cpu_manager.go:414] "containerMap: removing container" podUID="63cb9375-5f78-4f12-906e-9f6e6b274a96" containerName="revoke-keys"
Oct 31 13:19:13.947876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.947866    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63cb9375-5f78-4f12-906e-9f6e6b274a96" containerName="revoke-keys"
Oct 31 13:19:13.947962 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.947950    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="63cb9375-5f78-4f12-906e-9f6e6b274a96" containerName="revoke-keys"
Oct 31 13:19:13.952278 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.952259    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:13.958443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.958322    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:19:13.958622 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.958392    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:19:13.958622 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.958397    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:19:13.959030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.959010    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:19:13.961749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:13.961728    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"]
Oct 31 13:19:14.140621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.140567    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5pqh2\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.140621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.140634    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5pqh2\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.140914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.140744    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn4p9\" (UniqueName: \"kubernetes.io/projected/07d10be3-9a59-4528-9f8a-1fc29d912e1d-kube-api-access-tn4p9\") pod \"maas-api-revoke-sub-ae594979-5pqh2\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.241391 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.241301    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5pqh2\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.241391 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.241345    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5pqh2\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.241391 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.241367    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn4p9\" (UniqueName: \"kubernetes.io/projected/07d10be3-9a59-4528-9f8a-1fc29d912e1d-kube-api-access-tn4p9\") pod \"maas-api-revoke-sub-ae594979-5pqh2\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.242057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.242032    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-5pqh2\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.244013 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.243993    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-5pqh2\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.255565 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.255536    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn4p9\" (UniqueName: \"kubernetes.io/projected/07d10be3-9a59-4528-9f8a-1fc29d912e1d-kube-api-access-tn4p9\") pod \"maas-api-revoke-sub-ae594979-5pqh2\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.267196 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.267168    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:14.548505 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:14.548478    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"]
Oct 31 13:19:14.550316 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:19:14.550288    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07d10be3_9a59_4528_9f8a_1fc29d912e1d.slice/crio-56bb7edfe9592c3b658c3a06c05b12d2a22f3403507b824177537ce8a1d7c87b WatchSource:0}: Error finding container 56bb7edfe9592c3b658c3a06c05b12d2a22f3403507b824177537ce8a1d7c87b: Status 404 returned error can't find the container with id 56bb7edfe9592c3b658c3a06c05b12d2a22f3403507b824177537ce8a1d7c87b
Oct 31 13:19:15.255718 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:15.255672    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2" event={"ID":"07d10be3-9a59-4528-9f8a-1fc29d912e1d","Type":"ContainerStarted","Data":"8764eae399dfd25b52633ef4c1d83066a7a799dfcf18640dc1a44a5c865090b3"}
Oct 31 13:19:15.256135 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:15.255763    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2" event={"ID":"07d10be3-9a59-4528-9f8a-1fc29d912e1d","Type":"ContainerStarted","Data":"56bb7edfe9592c3b658c3a06c05b12d2a22f3403507b824177537ce8a1d7c87b"}
Oct 31 13:19:15.276678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:15.276624    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2" podStartSLOduration=2.276608953 podStartE2EDuration="2.276608953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:19:15.275352038 +0000 UTC m=+1134.691293880" watchObservedRunningTime="2026-10-31 13:19:15.276608953 +0000 UTC m=+1134.692550793"
Oct 31 13:19:16.271288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:16.271254    5787 generic.go:356] "Generic (PLEG): container finished" podID="07d10be3-9a59-4528-9f8a-1fc29d912e1d" containerID="8764eae399dfd25b52633ef4c1d83066a7a799dfcf18640dc1a44a5c865090b3" exitCode=0
Oct 31 13:19:16.271794 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:16.271343    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2" event={"ID":"07d10be3-9a59-4528-9f8a-1fc29d912e1d","Type":"ContainerDied","Data":"8764eae399dfd25b52633ef4c1d83066a7a799dfcf18640dc1a44a5c865090b3"}
Oct 31 13:19:17.409780 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.409747    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:17.575996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.575896    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-openshift-ca\") pod \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") "
Oct 31 13:19:17.576176 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.576026    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d10be3-9a59-4528-9f8a-1fc29d912e1d-kube-api-access-tn4p9\" (UniqueName: \"kubernetes.io/projected/07d10be3-9a59-4528-9f8a-1fc29d912e1d-kube-api-access-tn4p9\") pod \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") "
Oct 31 13:19:17.576176 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.576066    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-xks-ca\") pod \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\" (UID: \"07d10be3-9a59-4528-9f8a-1fc29d912e1d\") "
Oct 31 13:19:17.576372 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.576346    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-openshift-ca" pod "07d10be3-9a59-4528-9f8a-1fc29d912e1d" (UID: "07d10be3-9a59-4528-9f8a-1fc29d912e1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:19:17.578303 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.578272    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-xks-ca" pod "07d10be3-9a59-4528-9f8a-1fc29d912e1d" (UID: "07d10be3-9a59-4528-9f8a-1fc29d912e1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:19:17.578414 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.578307    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d10be3-9a59-4528-9f8a-1fc29d912e1d-kube-api-access-tn4p9" pod "07d10be3-9a59-4528-9f8a-1fc29d912e1d" (UID: "07d10be3-9a59-4528-9f8a-1fc29d912e1d"). InnerVolumeSpecName "kube-api-access-tn4p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:19:17.676764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.676682    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:19:17.676764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.676760    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn4p9\" (UniqueName: \"kubernetes.io/projected/07d10be3-9a59-4528-9f8a-1fc29d912e1d-kube-api-access-tn4p9\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:19:17.676764 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:17.676773    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07d10be3-9a59-4528-9f8a-1fc29d912e1d-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:19:18.245254 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:19:18.245221    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:19:18.285442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:18.285410    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"
Oct 31 13:19:18.285620 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:18.285444    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2" event={"ID":"07d10be3-9a59-4528-9f8a-1fc29d912e1d","Type":"ContainerDied","Data":"56bb7edfe9592c3b658c3a06c05b12d2a22f3403507b824177537ce8a1d7c87b"}
Oct 31 13:19:18.285620 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:18.285495    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56bb7edfe9592c3b658c3a06c05b12d2a22f3403507b824177537ce8a1d7c87b"
Oct 31 13:19:22.023178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.023136    5787 scope.go:122] "RemoveContainer" containerID="3d62ad61489657ed6122b2cca961ab7cd996b4ef5105c9df28c1fa60cb871890"
Oct 31 13:19:22.027490 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.027425    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"]
Oct 31 13:19:22.030521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.030496    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7428758a-m4k4g"]
Oct 31 13:19:22.034331 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.034312    5787 scope.go:122] "RemoveContainer" containerID="adda22546f1bfb77e2bdd69e2c3df04199c590c63f826e821ac88cb9570d83ea"
Oct 31 13:19:22.046560 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.046537    5787 scope.go:122] "RemoveContainer" containerID="482a072fca26f4232a9b7d8d85d9a0375de835a93b1d26b0143b664bc400e675"
Oct 31 13:19:22.056052 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.056032    5787 scope.go:122] "RemoveContainer" containerID="f56d4bb473b49af12b1bf1f070287c11966cde7ae3d686e2166214802ea0ec4e"
Oct 31 13:19:22.065077 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.065051    5787 scope.go:122] "RemoveContainer" containerID="185ee587f57664a2b0334cf86c1ba4dcceec3591922ac547406850c5e459a284"
Oct 31 13:19:22.073672 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.073654    5787 scope.go:122] "RemoveContainer" containerID="9e69b4ca314adf3766ed731c51dd80ba57f4b3e658db601d3cb7ad81c934455c"
Oct 31 13:19:22.082694 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.082674    5787 scope.go:122] "RemoveContainer" containerID="6b7ba812d80b6d8bb7a9e116afaaf9f5f523b82bf5fa7626a51ea1bd0cdd698e"
Oct 31 13:19:22.091639 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.091617    5787 scope.go:122] "RemoveContainer" containerID="bbff58ee74be716cb24e047d86334f9979eab153adac8503b89b4c2fd4f61899"
Oct 31 13:19:22.101759 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.101742    5787 scope.go:122] "RemoveContainer" containerID="fc5b3c28c7847c80a435c528edcff916248761f6cbd641560e14ecf0e7a0e92c"
Oct 31 13:19:22.110267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.110245    5787 scope.go:122] "RemoveContainer" containerID="e1e5e6ef1de3a63d7955946e385914cd1f20f8d8b4ffd3f5bb520e3557fd0184"
Oct 31 13:19:22.118801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.118785    5787 scope.go:122] "RemoveContainer" containerID="1cca89e296c01365980ef2e92a0ff1db52fbe801e61e7a83093aa1439607d252"
Oct 31 13:19:22.127153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.127136    5787 scope.go:122] "RemoveContainer" containerID="2569dd32ca6cffd5d2266eda18654caad9e6945dc296ed65aa00fa5a4b262b48"
Oct 31 13:19:22.135272 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.135253    5787 scope.go:122] "RemoveContainer" containerID="67f28dda93654a1801ea7a0669f8c5eafaf70a47224510a7201cf41eb2b457f8"
Oct 31 13:19:22.144603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.144583    5787 scope.go:122] "RemoveContainer" containerID="825784839eaaa097a801cc7bf8d4ccdfe4dfc877654c1b7a4dcd8a31a34b994b"
Oct 31 13:19:22.155549 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.155527    5787 scope.go:122] "RemoveContainer" containerID="a23c92c10f0a4afd0153ce50fbd229158bc48e2e4304f61ee4460decc98b4782"
Oct 31 13:19:22.164643 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.164623    5787 scope.go:122] "RemoveContainer" containerID="73f0a9ccd50efb14539d9dd72bc9181428bf34d8741ba5cf6a5065e8b7e55f36"
Oct 31 13:19:22.704822 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:22.704791    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7797c9-9794-4086-a59f-be440d6c72ad" path="/var/lib/kubelet/pods/0b7797c9-9794-4086-a59f-be440d6c72ad/volumes"
Oct 31 13:19:28.062730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:28.061761    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"]
Oct 31 13:19:28.062730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:28.061834    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"]
Oct 31 13:19:28.090732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:28.081759    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46cc47a8-pgzh9"]
Oct 31 13:19:28.094181 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:28.094105    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64dfbcc8-v9t2c"]
Oct 31 13:19:28.712802 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:28.712772    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d7253c9-9e35-449b-ac5a-73b9bfc583fa" path="/var/lib/kubelet/pods/8d7253c9-9e35-449b-ac5a-73b9bfc583fa/volumes"
Oct 31 13:19:28.713477 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:28.713458    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f07906e9-d37b-4cad-8296-c9104942cdbc" path="/var/lib/kubelet/pods/f07906e9-d37b-4cad-8296-c9104942cdbc/volumes"
Oct 31 13:19:32.199442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.199402    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"]
Oct 31 13:19:32.199898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.199856    5787 cpu_manager.go:414] "containerMap: removing container" podUID="07d10be3-9a59-4528-9f8a-1fc29d912e1d" containerName="revoke-keys"
Oct 31 13:19:32.199898 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.199868    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d10be3-9a59-4528-9f8a-1fc29d912e1d" containerName="revoke-keys"
Oct 31 13:19:32.199985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.199929    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d10be3-9a59-4528-9f8a-1fc29d912e1d" containerName="revoke-keys"
Oct 31 13:19:32.206577 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.206557    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.214789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.214764    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:19:32.214919 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.214862    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:19:32.214919 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.214909    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:19:32.215034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.215005    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:19:32.217150 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.217128    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"]
Oct 31 13:19:32.228614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.228544    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-sv4m6\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.228614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.228582    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wstpt\" (UniqueName: \"kubernetes.io/projected/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-kube-api-access-wstpt\") pod \"maas-api-revoke-sub-a4aef7d7-sv4m6\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.228816 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.228664    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-sv4m6\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.329489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.329459    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-sv4m6\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.329489 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.329492    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wstpt\" (UniqueName: \"kubernetes.io/projected/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-kube-api-access-wstpt\") pod \"maas-api-revoke-sub-a4aef7d7-sv4m6\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.329742 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.329640    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-sv4m6\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.330257 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.330238    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-sv4m6\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.331955 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.331927    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-sv4m6\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.343636 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.343615    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wstpt\" (UniqueName: \"kubernetes.io/projected/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-kube-api-access-wstpt\") pod \"maas-api-revoke-sub-a4aef7d7-sv4m6\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.521647 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.521555    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:32.805473 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:32.805443    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"]
Oct 31 13:19:32.807051 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:19:32.807022    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c57ffcb_8fd1_4ce1_a845_b83e4a5efe17.slice/crio-278c5b5c0003e4ee0cde4f4e88c78f9649d1d74a60492223bddd4b2e37d69a96 WatchSource:0}: Error finding container 278c5b5c0003e4ee0cde4f4e88c78f9649d1d74a60492223bddd4b2e37d69a96: Status 404 returned error can't find the container with id 278c5b5c0003e4ee0cde4f4e88c78f9649d1d74a60492223bddd4b2e37d69a96
Oct 31 13:19:33.410992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:33.410892    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6" event={"ID":"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17","Type":"ContainerStarted","Data":"f8d6a304931c915b5749ad00af5cbf8e15b78ddbc7a40c73067c81fc7399a7d0"}
Oct 31 13:19:33.410992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:33.410958    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6" event={"ID":"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17","Type":"ContainerStarted","Data":"278c5b5c0003e4ee0cde4f4e88c78f9649d1d74a60492223bddd4b2e37d69a96"}
Oct 31 13:19:33.434567 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:33.434060    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6" podStartSLOduration=1.4340431200000001 podStartE2EDuration="1.43404312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:19:33.433948216 +0000 UTC m=+1152.849890057" watchObservedRunningTime="2026-10-31 13:19:33.43404312 +0000 UTC m=+1152.849984960"
Oct 31 13:19:34.031781 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:34.031748    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"]
Oct 31 13:19:34.041364 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:34.041336    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28a55ca0-kms62"]
Oct 31 13:19:34.417959 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:34.417926    5787 generic.go:356] "Generic (PLEG): container finished" podID="8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17" containerID="f8d6a304931c915b5749ad00af5cbf8e15b78ddbc7a40c73067c81fc7399a7d0" exitCode=0
Oct 31 13:19:34.418394 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:34.418012    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6" event={"ID":"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17","Type":"ContainerDied","Data":"f8d6a304931c915b5749ad00af5cbf8e15b78ddbc7a40c73067c81fc7399a7d0"}
Oct 31 13:19:34.705675 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:34.705644    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="744e9587-802d-495d-b30e-cec079e3f3f8" path="/var/lib/kubelet/pods/744e9587-802d-495d-b30e-cec079e3f3f8/volumes"
Oct 31 13:19:35.560853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.560822    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:35.661920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.661889    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-openshift-ca\") pod \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") "
Oct 31 13:19:35.662122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.661989    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-kube-api-access-wstpt\" (UniqueName: \"kubernetes.io/projected/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-kube-api-access-wstpt\") pod \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") "
Oct 31 13:19:35.662196 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.662160    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-xks-ca\") pod \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\" (UID: \"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17\") "
Oct 31 13:19:35.662347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.662323    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-openshift-ca" pod "8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17" (UID: "8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:19:35.662524 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.662507    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:19:35.664235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.664213    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-kube-api-access-wstpt" pod "8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17" (UID: "8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17"). InnerVolumeSpecName "kube-api-access-wstpt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:19:35.664314 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.664297    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-xks-ca" pod "8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17" (UID: "8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:19:35.763293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.763250    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wstpt\" (UniqueName: \"kubernetes.io/projected/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-kube-api-access-wstpt\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:19:35.763293 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:35.763284    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:19:36.431473 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:36.431444    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"
Oct 31 13:19:36.431659 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:36.431486    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6" event={"ID":"8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17","Type":"ContainerDied","Data":"278c5b5c0003e4ee0cde4f4e88c78f9649d1d74a60492223bddd4b2e37d69a96"}
Oct 31 13:19:36.431659 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:36.431536    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="278c5b5c0003e4ee0cde4f4e88c78f9649d1d74a60492223bddd4b2e37d69a96"
Oct 31 13:19:40.027871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:40.027830    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"]
Oct 31 13:19:40.031312 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:40.031285    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ddf55a7f-zxsj7"]
Oct 31 13:19:40.704978 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:40.704945    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875d6f05-6e36-4c65-bea2-11c71f1997bb" path="/var/lib/kubelet/pods/875d6f05-6e36-4c65-bea2-11c71f1997bb/volumes"
Oct 31 13:19:48.252484 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:19:48.252446    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:19:53.132335 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.132300    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"]
Oct 31 13:19:53.132790 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.132775    5787 cpu_manager.go:414] "containerMap: removing container" podUID="8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17" containerName="revoke-keys"
Oct 31 13:19:53.132851 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.132791    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17" containerName="revoke-keys"
Oct 31 13:19:53.132895 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.132859    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17" containerName="revoke-keys"
Oct 31 13:19:53.136445 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.136424    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.143407 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.143381    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:19:53.144018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.143402    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:19:53.144018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.143410    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:19:53.144210 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.143504    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:19:53.146156 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.146130    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"]
Oct 31 13:19:53.247203 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.247166    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2hk\" (UniqueName: \"kubernetes.io/projected/65dc6323-857b-40bf-946c-a8edf854a44e-kube-api-access-7q2hk\") pod \"maas-api-revoke-sub-79805ca9-vj2bg\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.247387 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.247222    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vj2bg\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.247387 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.247293    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vj2bg\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.348136 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.348098    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2hk\" (UniqueName: \"kubernetes.io/projected/65dc6323-857b-40bf-946c-a8edf854a44e-kube-api-access-7q2hk\") pod \"maas-api-revoke-sub-79805ca9-vj2bg\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.348338 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.348149    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vj2bg\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.348338 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.348173    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vj2bg\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.349018 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.348942    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-vj2bg\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.350684 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.350662    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-vj2bg\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.361383 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.361360    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2hk\" (UniqueName: \"kubernetes.io/projected/65dc6323-857b-40bf-946c-a8edf854a44e-kube-api-access-7q2hk\") pod \"maas-api-revoke-sub-79805ca9-vj2bg\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.450461 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.450429    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:53.731967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:53.731938    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"]
Oct 31 13:19:53.733262 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:19:53.733235    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65dc6323_857b_40bf_946c_a8edf854a44e.slice/crio-0d7a7915929383d35690bc630378e6223c400ac7890218db11342f66351b6eb5 WatchSource:0}: Error finding container 0d7a7915929383d35690bc630378e6223c400ac7890218db11342f66351b6eb5: Status 404 returned error can't find the container with id 0d7a7915929383d35690bc630378e6223c400ac7890218db11342f66351b6eb5
Oct 31 13:19:54.548773 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:54.548741    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg" event={"ID":"65dc6323-857b-40bf-946c-a8edf854a44e","Type":"ContainerStarted","Data":"26b306e62956aaa0bfca2a60c489ffdabb1e2ac8362dd8b2bb13d95f6ab7fac1"}
Oct 31 13:19:54.549135 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:54.548808    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg" event={"ID":"65dc6323-857b-40bf-946c-a8edf854a44e","Type":"ContainerStarted","Data":"0d7a7915929383d35690bc630378e6223c400ac7890218db11342f66351b6eb5"}
Oct 31 13:19:55.557556 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.557517    5787 generic.go:356] "Generic (PLEG): container finished" podID="65dc6323-857b-40bf-946c-a8edf854a44e" containerID="26b306e62956aaa0bfca2a60c489ffdabb1e2ac8362dd8b2bb13d95f6ab7fac1" exitCode=0
Oct 31 13:19:55.557999 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.557598    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg" event={"ID":"65dc6323-857b-40bf-946c-a8edf854a44e","Type":"ContainerDied","Data":"26b306e62956aaa0bfca2a60c489ffdabb1e2ac8362dd8b2bb13d95f6ab7fac1"}
Oct 31 13:19:55.690264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.690239    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:55.770798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.770765    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-openshift-ca\") pod \"65dc6323-857b-40bf-946c-a8edf854a44e\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") "
Oct 31 13:19:55.770985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.770883    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-xks-ca\") pod \"65dc6323-857b-40bf-946c-a8edf854a44e\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") "
Oct 31 13:19:55.770985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.770912    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65dc6323-857b-40bf-946c-a8edf854a44e-kube-api-access-7q2hk\" (UniqueName: \"kubernetes.io/projected/65dc6323-857b-40bf-946c-a8edf854a44e-kube-api-access-7q2hk\") pod \"65dc6323-857b-40bf-946c-a8edf854a44e\" (UID: \"65dc6323-857b-40bf-946c-a8edf854a44e\") "
Oct 31 13:19:55.771242 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.771213    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-openshift-ca" pod "65dc6323-857b-40bf-946c-a8edf854a44e" (UID: "65dc6323-857b-40bf-946c-a8edf854a44e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:19:55.771348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.771328    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:19:55.773203 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.773181    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-xks-ca" pod "65dc6323-857b-40bf-946c-a8edf854a44e" (UID: "65dc6323-857b-40bf-946c-a8edf854a44e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:19:55.773283 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.773209    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65dc6323-857b-40bf-946c-a8edf854a44e-kube-api-access-7q2hk" pod "65dc6323-857b-40bf-946c-a8edf854a44e" (UID: "65dc6323-857b-40bf-946c-a8edf854a44e"). InnerVolumeSpecName "kube-api-access-7q2hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:19:55.872274 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.872191    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65dc6323-857b-40bf-946c-a8edf854a44e-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:19:55.872274 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:55.872218    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q2hk\" (UniqueName: \"kubernetes.io/projected/65dc6323-857b-40bf-946c-a8edf854a44e-kube-api-access-7q2hk\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:19:56.565490 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:56.565455    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"
Oct 31 13:19:56.565923 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:56.565487    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg" event={"ID":"65dc6323-857b-40bf-946c-a8edf854a44e","Type":"ContainerDied","Data":"0d7a7915929383d35690bc630378e6223c400ac7890218db11342f66351b6eb5"}
Oct 31 13:19:56.565923 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:56.565533    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d7a7915929383d35690bc630378e6223c400ac7890218db11342f66351b6eb5"
Oct 31 13:19:59.025344 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:59.025301    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"]
Oct 31 13:19:59.028689 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:19:59.028654    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-804d78ab-rhhww"]
Oct 31 13:20:00.704152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:00.704073    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="845451b4-9361-4c2b-a593-b7a68b6f6362" path="/var/lib/kubelet/pods/845451b4-9361-4c2b-a593-b7a68b6f6362/volumes"
Oct 31 13:20:02.027569 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:02.027532    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"]
Oct 31 13:20:02.031030 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:02.031002    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-745b7bde-t5frp"]
Oct 31 13:20:02.704355 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:02.704315    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="800893dc-ceaa-4010-87f6-4eaba5bc5aef" path="/var/lib/kubelet/pods/800893dc-ceaa-4010-87f6-4eaba5bc5aef/volumes"
Oct 31 13:20:08.050732 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:08.049764    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"]
Oct 31 13:20:08.074731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:08.073413    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dac9536-fq5d5"]
Oct 31 13:20:08.709933 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:08.709895    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1925a03-bd6a-437b-aa44-5f8c2741cd15" path="/var/lib/kubelet/pods/a1925a03-bd6a-437b-aa44-5f8c2741cd15/volumes"
Oct 31 13:20:10.932025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.931991    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"]
Oct 31 13:20:10.932439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.932423    5787 cpu_manager.go:414] "containerMap: removing container" podUID="65dc6323-857b-40bf-946c-a8edf854a44e" containerName="revoke-keys"
Oct 31 13:20:10.932439 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.932438    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65dc6323-857b-40bf-946c-a8edf854a44e" containerName="revoke-keys"
Oct 31 13:20:10.932534 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.932522    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="65dc6323-857b-40bf-946c-a8edf854a44e" containerName="revoke-keys"
Oct 31 13:20:10.937411 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.937385    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:10.944095 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.944038    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:20:10.944356 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.944083    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:20:10.944356 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.944169    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:20:10.944356 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.944216    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:20:10.945939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:10.945914    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"]
Oct 31 13:20:11.046523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.046490    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rklgg\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.046737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.046565    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnhz5\" (UniqueName: \"kubernetes.io/projected/af2e4541-14bb-4453-9429-441c6ae06326-kube-api-access-xnhz5\") pod \"maas-api-revoke-sub-0b7e8edb-rklgg\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.046737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.046669    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rklgg\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.147533 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.147491    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnhz5\" (UniqueName: \"kubernetes.io/projected/af2e4541-14bb-4453-9429-441c6ae06326-kube-api-access-xnhz5\") pod \"maas-api-revoke-sub-0b7e8edb-rklgg\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.147804 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.147541    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rklgg\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.147804 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.147628    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rklgg\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.148415 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.148390    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rklgg\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.150265 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.150245    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rklgg\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.162104 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.162078    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnhz5\" (UniqueName: \"kubernetes.io/projected/af2e4541-14bb-4453-9429-441c6ae06326-kube-api-access-xnhz5\") pod \"maas-api-revoke-sub-0b7e8edb-rklgg\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.255897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.255861    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:11.533559 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.533530    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"]
Oct 31 13:20:11.535825 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:20:11.535791    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2e4541_14bb_4453_9429_441c6ae06326.slice/crio-3f2aed4d9545a3cfce52a473d8c2a2a6e2427f561c3b0d0353e6cd9a128d521c WatchSource:0}: Error finding container 3f2aed4d9545a3cfce52a473d8c2a2a6e2427f561c3b0d0353e6cd9a128d521c: Status 404 returned error can't find the container with id 3f2aed4d9545a3cfce52a473d8c2a2a6e2427f561c3b0d0353e6cd9a128d521c
Oct 31 13:20:11.695331 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:11.695293    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg" event={"ID":"af2e4541-14bb-4453-9429-441c6ae06326","Type":"ContainerStarted","Data":"3f2aed4d9545a3cfce52a473d8c2a2a6e2427f561c3b0d0353e6cd9a128d521c"}
Oct 31 13:20:12.026138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:12.026090    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"]
Oct 31 13:20:12.028468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:12.028444    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6f77003-5w5mr"]
Oct 31 13:20:12.704530 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:12.704493    5787 generic.go:356] "Generic (PLEG): container finished" podID="af2e4541-14bb-4453-9429-441c6ae06326" containerID="8706ed8520e40ef2965746e8ee73ed2f4a4eb7b7e286e028d421912322d68b37" exitCode=0
Oct 31 13:20:12.705693 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:12.705671    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371beb92-8f45-480b-bd39-6bf4d1d9d987" path="/var/lib/kubelet/pods/371beb92-8f45-480b-bd39-6bf4d1d9d987/volumes"
Oct 31 13:20:12.705975 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:12.705959    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg" event={"ID":"af2e4541-14bb-4453-9429-441c6ae06326","Type":"ContainerDied","Data":"8706ed8520e40ef2965746e8ee73ed2f4a4eb7b7e286e028d421912322d68b37"}
Oct 31 13:20:13.034202 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.034114    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"]
Oct 31 13:20:13.042994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.042963    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fdd5189-vbqv5"]
Oct 31 13:20:13.845094 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.845070    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:13.871452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.871418    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-openshift-ca\") pod \"af2e4541-14bb-4453-9429-441c6ae06326\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") "
Oct 31 13:20:13.871626 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.871475    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-xks-ca\") pod \"af2e4541-14bb-4453-9429-441c6ae06326\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") "
Oct 31 13:20:13.871626 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.871513    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af2e4541-14bb-4453-9429-441c6ae06326-kube-api-access-xnhz5\" (UniqueName: \"kubernetes.io/projected/af2e4541-14bb-4453-9429-441c6ae06326-kube-api-access-xnhz5\") pod \"af2e4541-14bb-4453-9429-441c6ae06326\" (UID: \"af2e4541-14bb-4453-9429-441c6ae06326\") "
Oct 31 13:20:13.871870 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.871841    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-openshift-ca" pod "af2e4541-14bb-4453-9429-441c6ae06326" (UID: "af2e4541-14bb-4453-9429-441c6ae06326"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:20:13.873857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.873832    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af2e4541-14bb-4453-9429-441c6ae06326-kube-api-access-xnhz5" pod "af2e4541-14bb-4453-9429-441c6ae06326" (UID: "af2e4541-14bb-4453-9429-441c6ae06326"). InnerVolumeSpecName "kube-api-access-xnhz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:20:13.873975 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.873909    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-xks-ca" pod "af2e4541-14bb-4453-9429-441c6ae06326" (UID: "af2e4541-14bb-4453-9429-441c6ae06326"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:20:13.972609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.972529    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:13.972609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.972556    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af2e4541-14bb-4453-9429-441c6ae06326-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:13.972609 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:13.972566    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnhz5\" (UniqueName: \"kubernetes.io/projected/af2e4541-14bb-4453-9429-441c6ae06326-kube-api-access-xnhz5\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:14.704061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:14.704021    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b95431-cb3c-458b-b585-4afa8c808590" path="/var/lib/kubelet/pods/65b95431-cb3c-458b-b585-4afa8c808590/volumes"
Oct 31 13:20:14.723287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:14.723262    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"
Oct 31 13:20:14.723435 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:14.723286    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg" event={"ID":"af2e4541-14bb-4453-9429-441c6ae06326","Type":"ContainerDied","Data":"3f2aed4d9545a3cfce52a473d8c2a2a6e2427f561c3b0d0353e6cd9a128d521c"}
Oct 31 13:20:14.723435 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:14.723353    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f2aed4d9545a3cfce52a473d8c2a2a6e2427f561c3b0d0353e6cd9a128d521c"
Oct 31 13:20:16.029002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:16.028968    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"]
Oct 31 13:20:16.033116 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:16.033083    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db78c405-nwdvt"]
Oct 31 13:20:16.704021 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:16.703988    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb9a0f4-b481-496b-b905-e9667799c175" path="/var/lib/kubelet/pods/8eb9a0f4-b481-496b-b905-e9667799c175/volumes"
Oct 31 13:20:17.027829 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:17.027736    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"]
Oct 31 13:20:17.030130 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:17.030106    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-392c74b1-chkf9"]
Oct 31 13:20:18.245648 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:20:18.245606    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:20:18.704729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:18.704680    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d0db60-46fc-4929-bcae-f0bb9f2e587b" path="/var/lib/kubelet/pods/e1d0db60-46fc-4929-bcae-f0bb9f2e587b/volumes"
Oct 31 13:20:21.624879 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.624831    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"]
Oct 31 13:20:21.625451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.625430    5787 cpu_manager.go:414] "containerMap: removing container" podUID="af2e4541-14bb-4453-9429-441c6ae06326" containerName="revoke-keys"
Oct 31 13:20:21.625548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.625453    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af2e4541-14bb-4453-9429-441c6ae06326" containerName="revoke-keys"
Oct 31 13:20:21.625606 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.625597    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="af2e4541-14bb-4453-9429-441c6ae06326" containerName="revoke-keys"
Oct 31 13:20:21.630176 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.630154    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.636528 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.636507    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:20:21.636796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.636572    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:20:21.639436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.638570    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:20:21.639436 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.638996    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:20:21.640228 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.640176    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"]
Oct 31 13:20:21.745268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.745226    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2rzx\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.745268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.745276    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhxmk\" (UniqueName: \"kubernetes.io/projected/5ea384fb-aead-4143-a3d0-ba6582e10c1a-kube-api-access-rhxmk\") pod \"maas-api-revoke-sub-3d4156f1-s2rzx\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.745494 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.745433    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2rzx\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.845977 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.845941    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2rzx\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.846178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.846016    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2rzx\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.846178 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.846065    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhxmk\" (UniqueName: \"kubernetes.io/projected/5ea384fb-aead-4143-a3d0-ba6582e10c1a-kube-api-access-rhxmk\") pod \"maas-api-revoke-sub-3d4156f1-s2rzx\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.846644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.846625    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2rzx\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.848427 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.848404    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s2rzx\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.859783 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.859752    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhxmk\" (UniqueName: \"kubernetes.io/projected/5ea384fb-aead-4143-a3d0-ba6582e10c1a-kube-api-access-rhxmk\") pod \"maas-api-revoke-sub-3d4156f1-s2rzx\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:21.946097 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:21.946063    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:22.245329 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:22.245234    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"]
Oct 31 13:20:22.250859 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:20:22.250828    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea384fb_aead_4143_a3d0_ba6582e10c1a.slice/crio-d5a42fce8a9e0ad00c5109e6d3d49b6ed4c75e953aca4bdddc99acf3cb0744b5 WatchSource:0}: Error finding container d5a42fce8a9e0ad00c5109e6d3d49b6ed4c75e953aca4bdddc99acf3cb0744b5: Status 404 returned error can't find the container with id d5a42fce8a9e0ad00c5109e6d3d49b6ed4c75e953aca4bdddc99acf3cb0744b5
Oct 31 13:20:22.256041 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:22.256014    5787 scope.go:122] "RemoveContainer" containerID="969dc7621a11288b6903fe19bdefdd03198126b299026d53397759bae36bf102"
Oct 31 13:20:22.271183 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:22.271148    5787 scope.go:122] "RemoveContainer" containerID="a85ecf9804f6d9baa0493f769d8f30c8027fa314822463e013f412297813d408"
Oct 31 13:20:22.283839 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:22.283808    5787 scope.go:122] "RemoveContainer" containerID="0cb70b12161ff86d85b587f2dafd5a3a957c6fc4be806f84a0e9fb0fdd56e143"
Oct 31 13:20:22.296920 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:22.296887    5787 scope.go:122] "RemoveContainer" containerID="3640241891b9c4525a965ee8e0fc579d6a4a55ef6c4f8e1ed1b9da4e19bf4b5f"
Oct 31 13:20:22.774684 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:22.774642    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx" event={"ID":"5ea384fb-aead-4143-a3d0-ba6582e10c1a","Type":"ContainerStarted","Data":"1460ee20aeca4731ecf658588a258a935710513c698313bb9e76aee61c293d6c"}
Oct 31 13:20:22.775182 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:22.774698    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx" event={"ID":"5ea384fb-aead-4143-a3d0-ba6582e10c1a","Type":"ContainerStarted","Data":"d5a42fce8a9e0ad00c5109e6d3d49b6ed4c75e953aca4bdddc99acf3cb0744b5"}
Oct 31 13:20:22.794758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:22.794630    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx" podStartSLOduration=1.794616012 podStartE2EDuration="1.794616012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:20:22.79396379 +0000 UTC m=+1202.209905631" watchObservedRunningTime="2026-10-31 13:20:22.794616012 +0000 UTC m=+1202.210557853"
Oct 31 13:20:23.783236 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:23.783199    5787 generic.go:356] "Generic (PLEG): container finished" podID="5ea384fb-aead-4143-a3d0-ba6582e10c1a" containerID="1460ee20aeca4731ecf658588a258a935710513c698313bb9e76aee61c293d6c" exitCode=0
Oct 31 13:20:23.783673 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:23.783293    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx" event={"ID":"5ea384fb-aead-4143-a3d0-ba6582e10c1a","Type":"ContainerDied","Data":"1460ee20aeca4731ecf658588a258a935710513c698313bb9e76aee61c293d6c"}
Oct 31 13:20:24.925749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:24.925697    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:24.977937 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:24.977899    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-xks-ca\") pod \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") "
Oct 31 13:20:24.978121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:24.977999    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-openshift-ca\") pod \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") "
Oct 31 13:20:24.978186 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:24.978157    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea384fb-aead-4143-a3d0-ba6582e10c1a-kube-api-access-rhxmk\" (UniqueName: \"kubernetes.io/projected/5ea384fb-aead-4143-a3d0-ba6582e10c1a-kube-api-access-rhxmk\") pod \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\" (UID: \"5ea384fb-aead-4143-a3d0-ba6582e10c1a\") "
Oct 31 13:20:24.978381 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:24.978357    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-openshift-ca" pod "5ea384fb-aead-4143-a3d0-ba6582e10c1a" (UID: "5ea384fb-aead-4143-a3d0-ba6582e10c1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:20:24.978548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:24.978526    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:24.980295 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:24.980267    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-xks-ca" pod "5ea384fb-aead-4143-a3d0-ba6582e10c1a" (UID: "5ea384fb-aead-4143-a3d0-ba6582e10c1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:20:24.980397 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:24.980328    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea384fb-aead-4143-a3d0-ba6582e10c1a-kube-api-access-rhxmk" pod "5ea384fb-aead-4143-a3d0-ba6582e10c1a" (UID: "5ea384fb-aead-4143-a3d0-ba6582e10c1a"). InnerVolumeSpecName "kube-api-access-rhxmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:20:25.079789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:25.079671    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ea384fb-aead-4143-a3d0-ba6582e10c1a-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:25.079789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:25.079733    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhxmk\" (UniqueName: \"kubernetes.io/projected/5ea384fb-aead-4143-a3d0-ba6582e10c1a-kube-api-access-rhxmk\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:25.797316 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:25.797284    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"
Oct 31 13:20:25.797499 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:25.797319    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx" event={"ID":"5ea384fb-aead-4143-a3d0-ba6582e10c1a","Type":"ContainerDied","Data":"d5a42fce8a9e0ad00c5109e6d3d49b6ed4c75e953aca4bdddc99acf3cb0744b5"}
Oct 31 13:20:25.797499 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:25.797373    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5a42fce8a9e0ad00c5109e6d3d49b6ed4c75e953aca4bdddc99acf3cb0744b5"
Oct 31 13:20:28.031395 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:28.031358    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"]
Oct 31 13:20:28.035152 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:28.035124    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b62cb9-nczgb"]
Oct 31 13:20:28.704552 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:28.704520    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0729f58a-7f4d-4966-90ac-796c49ade823" path="/var/lib/kubelet/pods/0729f58a-7f4d-4966-90ac-796c49ade823/volumes"
Oct 31 13:20:32.024577 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:32.024538    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"]
Oct 31 13:20:32.028551 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:32.028520    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"]
Oct 31 13:20:32.030938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:32.030912    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9bfd38d-6jhv2"]
Oct 31 13:20:32.032678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:32.032657    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c330c431-zg9lr"]
Oct 31 13:20:32.705163 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:32.705128    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6460e2c0-e9ba-4943-aee0-c06f74ede22d" path="/var/lib/kubelet/pods/6460e2c0-e9ba-4943-aee0-c06f74ede22d/volumes"
Oct 31 13:20:32.705534 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:32.705521    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efde1a18-f86c-4247-bd41-9ac23d244500" path="/var/lib/kubelet/pods/efde1a18-f86c-4247-bd41-9ac23d244500/volumes"
Oct 31 13:20:34.026049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:34.026010    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"]
Oct 31 13:20:34.031414 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:34.031385    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14d852eb-jb6c2"]
Oct 31 13:20:34.709520 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:34.709484    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a2fa8ce-cfe1-4226-9825-25365b18e0bf" path="/var/lib/kubelet/pods/0a2fa8ce-cfe1-4226-9825-25365b18e0bf/volumes"
Oct 31 13:20:37.026443 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:37.026407    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"]
Oct 31 13:20:37.032918 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:37.032883    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2eb7c570-vjg47"]
Oct 31 13:20:38.703864 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:38.703828    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582d711c-00e3-4c13-922d-ed2e924a7eae" path="/var/lib/kubelet/pods/582d711c-00e3-4c13-922d-ed2e924a7eae/volumes"
Oct 31 13:20:42.973909 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:42.973864    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"]
Oct 31 13:20:42.979878 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:42.979847    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s2rzx"]
Oct 31 13:20:43.001812 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.001777    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"]
Oct 31 13:20:43.002343 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.002323    5787 cpu_manager.go:414] "containerMap: removing container" podUID="5ea384fb-aead-4143-a3d0-ba6582e10c1a" containerName="revoke-keys"
Oct 31 13:20:43.002463 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.002344    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea384fb-aead-4143-a3d0-ba6582e10c1a" containerName="revoke-keys"
Oct 31 13:20:43.002523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.002468    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea384fb-aead-4143-a3d0-ba6582e10c1a" containerName="revoke-keys"
Oct 31 13:20:43.005774 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.005754    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.013730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.013679    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:20:43.013918 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.013892    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:20:43.014006 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.013928    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:20:43.014122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.014105    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:20:43.016125 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.016099    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"]
Oct 31 13:20:43.109326 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.109292    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gndcb\" (UniqueName: \"kubernetes.io/projected/57a98eec-6227-4e17-af10-c313dd77f884-kube-api-access-gndcb\") pod \"maas-api-revoke-sub-3d4156f1-kjtx9\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.109326 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.109330    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjtx9\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.109542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.109476    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjtx9\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.210371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.210329    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gndcb\" (UniqueName: \"kubernetes.io/projected/57a98eec-6227-4e17-af10-c313dd77f884-kube-api-access-gndcb\") pod \"maas-api-revoke-sub-3d4156f1-kjtx9\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.210371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.210369    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjtx9\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.210596 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.210430    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjtx9\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.211207 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.211189    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjtx9\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.212945 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.212926    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-kjtx9\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.224062 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.224006    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gndcb\" (UniqueName: \"kubernetes.io/projected/57a98eec-6227-4e17-af10-c313dd77f884-kube-api-access-gndcb\") pod \"maas-api-revoke-sub-3d4156f1-kjtx9\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.320861 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.320817    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:43.597640 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.597615    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"]
Oct 31 13:20:43.599990 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:20:43.599964    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a98eec_6227_4e17_af10_c313dd77f884.slice/crio-4dfec4e12b00379a13c25de7b2c14b1a112797bc6bf5e16fb74395b0b0550b20 WatchSource:0}: Error finding container 4dfec4e12b00379a13c25de7b2c14b1a112797bc6bf5e16fb74395b0b0550b20: Status 404 returned error can't find the container with id 4dfec4e12b00379a13c25de7b2c14b1a112797bc6bf5e16fb74395b0b0550b20
Oct 31 13:20:43.964792 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.964753    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9" event={"ID":"57a98eec-6227-4e17-af10-c313dd77f884","Type":"ContainerStarted","Data":"df7f1de005d37e8291e78d1174494e3e862bd8fac36cdc4469ef6a58cb3a76fa"}
Oct 31 13:20:43.965025 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.964810    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9" event={"ID":"57a98eec-6227-4e17-af10-c313dd77f884","Type":"ContainerStarted","Data":"4dfec4e12b00379a13c25de7b2c14b1a112797bc6bf5e16fb74395b0b0550b20"}
Oct 31 13:20:43.986953 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:43.986904    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9" podStartSLOduration=1.986890978 podStartE2EDuration="1.986890978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:20:43.985406164 +0000 UTC m=+1223.401348005" watchObservedRunningTime="2026-10-31 13:20:43.986890978 +0000 UTC m=+1223.402832827"
Oct 31 13:20:44.704317 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:44.704284    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea384fb-aead-4143-a3d0-ba6582e10c1a" path="/var/lib/kubelet/pods/5ea384fb-aead-4143-a3d0-ba6582e10c1a/volumes"
Oct 31 13:20:44.972483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:44.972398    5787 generic.go:356] "Generic (PLEG): container finished" podID="57a98eec-6227-4e17-af10-c313dd77f884" containerID="df7f1de005d37e8291e78d1174494e3e862bd8fac36cdc4469ef6a58cb3a76fa" exitCode=0
Oct 31 13:20:44.972483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:44.972448    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9" event={"ID":"57a98eec-6227-4e17-af10-c313dd77f884","Type":"ContainerDied","Data":"df7f1de005d37e8291e78d1174494e3e862bd8fac36cdc4469ef6a58cb3a76fa"}
Oct 31 13:20:46.116981 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.116955    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:46.240288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.240180    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-xks-ca\") pod \"57a98eec-6227-4e17-af10-c313dd77f884\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") "
Oct 31 13:20:46.240288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.240235    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a98eec-6227-4e17-af10-c313dd77f884-kube-api-access-gndcb\" (UniqueName: \"kubernetes.io/projected/57a98eec-6227-4e17-af10-c313dd77f884-kube-api-access-gndcb\") pod \"57a98eec-6227-4e17-af10-c313dd77f884\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") "
Oct 31 13:20:46.240558 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.240411    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-openshift-ca\") pod \"57a98eec-6227-4e17-af10-c313dd77f884\" (UID: \"57a98eec-6227-4e17-af10-c313dd77f884\") "
Oct 31 13:20:46.240779 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.240755    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-openshift-ca" pod "57a98eec-6227-4e17-af10-c313dd77f884" (UID: "57a98eec-6227-4e17-af10-c313dd77f884"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:20:46.242452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.242425    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-xks-ca" pod "57a98eec-6227-4e17-af10-c313dd77f884" (UID: "57a98eec-6227-4e17-af10-c313dd77f884"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:20:46.242526 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.242448    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a98eec-6227-4e17-af10-c313dd77f884-kube-api-access-gndcb" pod "57a98eec-6227-4e17-af10-c313dd77f884" (UID: "57a98eec-6227-4e17-af10-c313dd77f884"). InnerVolumeSpecName "kube-api-access-gndcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:20:46.341778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.341735    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:46.341778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.341772    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/57a98eec-6227-4e17-af10-c313dd77f884-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:46.341778 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.341783    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gndcb\" (UniqueName: \"kubernetes.io/projected/57a98eec-6227-4e17-af10-c313dd77f884-kube-api-access-gndcb\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:20:46.986963 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.986879    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"
Oct 31 13:20:46.986963 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.986909    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9" event={"ID":"57a98eec-6227-4e17-af10-c313dd77f884","Type":"ContainerDied","Data":"4dfec4e12b00379a13c25de7b2c14b1a112797bc6bf5e16fb74395b0b0550b20"}
Oct 31 13:20:46.987176 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:46.986965    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dfec4e12b00379a13c25de7b2c14b1a112797bc6bf5e16fb74395b0b0550b20"
Oct 31 13:20:47.029487 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:47.029447    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"]
Oct 31 13:20:47.033942 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:47.033913    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-524e5f43-k2chm"]
Oct 31 13:20:48.027228 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:48.027193    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"]
Oct 31 13:20:48.030483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:48.030458    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea26cbb-2clrl"]
Oct 31 13:20:48.245947 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:20:48.245914    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:20:48.704789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:48.704759    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a835cb0-8e7f-4ca7-943d-1813763f3a36" path="/var/lib/kubelet/pods/3a835cb0-8e7f-4ca7-943d-1813763f3a36/volumes"
Oct 31 13:20:48.705213 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:48.705198    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d013b999-e4c4-41df-9c4f-961babdabcac" path="/var/lib/kubelet/pods/d013b999-e4c4-41df-9c4f-961babdabcac/volumes"
Oct 31 13:20:49.026917 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:49.026821    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"]
Oct 31 13:20:49.029606 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:49.029575    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e7d1149-c65cf"]
Oct 31 13:20:50.705198 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:50.705158    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e" path="/var/lib/kubelet/pods/bc6aa5d4-ed8f-4e81-a067-fc98f32e8a1e/volumes"
Oct 31 13:20:59.026454 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:59.026413    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"]
Oct 31 13:20:59.029508 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:59.028495    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20d70537-9klbf"]
Oct 31 13:20:59.031181 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:59.031154    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"]
Oct 31 13:20:59.032688 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:59.032669    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"]
Oct 31 13:20:59.038033 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:59.037998    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9400ff4e-p4tgr"]
Oct 31 13:20:59.039423 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:20:59.039402    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0add37c5-br6lt"]
Oct 31 13:21:00.704468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:00.704432    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aaeb079-0a80-40c4-bb62-a9767bc80376" path="/var/lib/kubelet/pods/0aaeb079-0a80-40c4-bb62-a9767bc80376/volumes"
Oct 31 13:21:00.704892 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:00.704831    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e1cdfd-3a07-40bc-958d-874818c2e834" path="/var/lib/kubelet/pods/84e1cdfd-3a07-40bc-958d-874818c2e834/volumes"
Oct 31 13:21:00.705185 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:00.705173    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a859a8-afaa-458f-831b-7e549ce27426" path="/var/lib/kubelet/pods/c3a859a8-afaa-458f-831b-7e549ce27426/volumes"
Oct 31 13:21:06.026657 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:06.026625    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"]
Oct 31 13:21:06.029733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:06.029689    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"]
Oct 31 13:21:06.032299 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:06.032274    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db963afe-4c5pd"]
Oct 31 13:21:06.033758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:06.033728    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf668b65-j2xt6"]
Oct 31 13:21:06.709939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:06.709900    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa" path="/var/lib/kubelet/pods/3f701b62-f9ae-4a6a-9313-ecbdf3fe52fa/volumes"
Oct 31 13:21:06.710403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:06.710381    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa8967d-a072-4b5a-8322-0769b0df3578" path="/var/lib/kubelet/pods/cfa8967d-a072-4b5a-8322-0769b0df3578/volumes"
Oct 31 13:21:09.990731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:09.989967    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-rz8cm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:21:09.990731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:09.990082    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-rz8cm" podUID="d95c8381-0aff-448c-9422-d3693062872f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:21:14.908960 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.908921    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"]
Oct 31 13:21:14.909480 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.909460    5787 cpu_manager.go:414] "containerMap: removing container" podUID="57a98eec-6227-4e17-af10-c313dd77f884" containerName="revoke-keys"
Oct 31 13:21:14.909531 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.909483    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a98eec-6227-4e17-af10-c313dd77f884" containerName="revoke-keys"
Oct 31 13:21:14.909585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.909552    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a98eec-6227-4e17-af10-c313dd77f884" containerName="revoke-keys"
Oct 31 13:21:14.912719 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.912688    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:14.920919 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.920585    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:21:14.920919 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.920591    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:21:14.920919 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.920598    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:21:14.920919 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.920655    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:21:14.922312 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:14.922284    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"]
Oct 31 13:21:15.041481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.041437    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxnfx\" (UniqueName: \"kubernetes.io/projected/ede0ed7e-5c00-45af-b071-6bd7984a100c-kube-api-access-wxnfx\") pod \"maas-api-revoke-sub-e2fc9c5b-k6zz6\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.041481 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.041484    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-k6zz6\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.041737 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.041552    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-k6zz6\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.142457 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.142413    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxnfx\" (UniqueName: \"kubernetes.io/projected/ede0ed7e-5c00-45af-b071-6bd7984a100c-kube-api-access-wxnfx\") pod \"maas-api-revoke-sub-e2fc9c5b-k6zz6\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.142457 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.142460    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-k6zz6\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.142771 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.142495    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-k6zz6\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.143141 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.143118    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-k6zz6\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.145086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.145067    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-k6zz6\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.155758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.155735    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxnfx\" (UniqueName: \"kubernetes.io/projected/ede0ed7e-5c00-45af-b071-6bd7984a100c-kube-api-access-wxnfx\") pod \"maas-api-revoke-sub-e2fc9c5b-k6zz6\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.228771 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.228700    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:15.507362 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:15.507333    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"]
Oct 31 13:21:15.509981 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:21:15.509953    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede0ed7e_5c00_45af_b071_6bd7984a100c.slice/crio-f8b1b1ac1fd8811d336384491bef3412a282b4720f99609386654a8098609711 WatchSource:0}: Error finding container f8b1b1ac1fd8811d336384491bef3412a282b4720f99609386654a8098609711: Status 404 returned error can't find the container with id f8b1b1ac1fd8811d336384491bef3412a282b4720f99609386654a8098609711
Oct 31 13:21:16.196211 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:16.196167    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6" event={"ID":"ede0ed7e-5c00-45af-b071-6bd7984a100c","Type":"ContainerStarted","Data":"24590cc19d3f60d92389d72c83c4af8c55103953f0aafdc9e76b732ee1c8de95"}
Oct 31 13:21:16.196677 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:16.196232    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6" event={"ID":"ede0ed7e-5c00-45af-b071-6bd7984a100c","Type":"ContainerStarted","Data":"f8b1b1ac1fd8811d336384491bef3412a282b4720f99609386654a8098609711"}
Oct 31 13:21:16.218114 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:16.218052    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6" podStartSLOduration=2.218037243 podStartE2EDuration="2.218037243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:16.216445312 +0000 UTC m=+1255.632387165" watchObservedRunningTime="2026-10-31 13:21:16.218037243 +0000 UTC m=+1255.633979083"
Oct 31 13:21:17.203632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:17.203594    5787 generic.go:356] "Generic (PLEG): container finished" podID="ede0ed7e-5c00-45af-b071-6bd7984a100c" containerID="24590cc19d3f60d92389d72c83c4af8c55103953f0aafdc9e76b732ee1c8de95" exitCode=0
Oct 31 13:21:17.204082 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:17.203689    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6" event={"ID":"ede0ed7e-5c00-45af-b071-6bd7984a100c","Type":"ContainerDied","Data":"24590cc19d3f60d92389d72c83c4af8c55103953f0aafdc9e76b732ee1c8de95"}
Oct 31 13:21:18.247682 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:21:18.247641    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:21:18.344049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.344021    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:18.477234 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.477131    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-xks-ca\") pod \"ede0ed7e-5c00-45af-b071-6bd7984a100c\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") "
Oct 31 13:21:18.477234 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.477189    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede0ed7e-5c00-45af-b071-6bd7984a100c-kube-api-access-wxnfx\" (UniqueName: \"kubernetes.io/projected/ede0ed7e-5c00-45af-b071-6bd7984a100c-kube-api-access-wxnfx\") pod \"ede0ed7e-5c00-45af-b071-6bd7984a100c\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") "
Oct 31 13:21:18.477234 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.477238    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-openshift-ca\") pod \"ede0ed7e-5c00-45af-b071-6bd7984a100c\" (UID: \"ede0ed7e-5c00-45af-b071-6bd7984a100c\") "
Oct 31 13:21:18.477639 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.477613    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-openshift-ca" pod "ede0ed7e-5c00-45af-b071-6bd7984a100c" (UID: "ede0ed7e-5c00-45af-b071-6bd7984a100c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:21:18.479501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.479478    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-xks-ca" pod "ede0ed7e-5c00-45af-b071-6bd7984a100c" (UID: "ede0ed7e-5c00-45af-b071-6bd7984a100c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:21:18.479578 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.479481    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede0ed7e-5c00-45af-b071-6bd7984a100c-kube-api-access-wxnfx" pod "ede0ed7e-5c00-45af-b071-6bd7984a100c" (UID: "ede0ed7e-5c00-45af-b071-6bd7984a100c"). InnerVolumeSpecName "kube-api-access-wxnfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:21:18.578357 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.578317    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:18.578357 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.578348    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxnfx\" (UniqueName: \"kubernetes.io/projected/ede0ed7e-5c00-45af-b071-6bd7984a100c-kube-api-access-wxnfx\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:18.578357 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:18.578358    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ede0ed7e-5c00-45af-b071-6bd7984a100c-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:19.217306 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:19.217274    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"
Oct 31 13:21:19.217512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:19.217280    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6" event={"ID":"ede0ed7e-5c00-45af-b071-6bd7984a100c","Type":"ContainerDied","Data":"f8b1b1ac1fd8811d336384491bef3412a282b4720f99609386654a8098609711"}
Oct 31 13:21:19.217512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:19.217409    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8b1b1ac1fd8811d336384491bef3412a282b4720f99609386654a8098609711"
Oct 31 13:21:22.345507 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.345470    5787 scope.go:122] "RemoveContainer" containerID="97d02905484e588cbf593ee1477ce42a60cdcefabd4c88e3de6e109f4978ece7"
Oct 31 13:21:22.355243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.355220    5787 scope.go:122] "RemoveContainer" containerID="8a436775f6751def3060dde796d2b2d051e81985b40e31dbd22788a3719e1ab7"
Oct 31 13:21:22.365249 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.365226    5787 scope.go:122] "RemoveContainer" containerID="6bdead7a24befaa37918e5b6014dbf8c972e0081034a0bc0974b304346880cf0"
Oct 31 13:21:22.374058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.374038    5787 scope.go:122] "RemoveContainer" containerID="c45b09a8fc885dbec0f1ba8461671cc80d9cbe52673f8e65e0b5317ede5cdcd6"
Oct 31 13:21:22.382642 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.382623    5787 scope.go:122] "RemoveContainer" containerID="4687dcdd0eef1ae80bbc8ab46afe7d1c2403c9db4a079916c186120949b20dca"
Oct 31 13:21:22.391578 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.391553    5787 scope.go:122] "RemoveContainer" containerID="9f30a3db4fbe27c09e662bd697da5adaa4ded1b0d6329f05ba8ac928f743f823"
Oct 31 13:21:22.399871 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.399851    5787 scope.go:122] "RemoveContainer" containerID="d70b77e1fc84307ee70b0805ed4cb8fdf348cf15ec2813297454e26b8607e5d5"
Oct 31 13:21:22.408108 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.408087    5787 scope.go:122] "RemoveContainer" containerID="12c87682066575bb372f6c1549538397966721dd311fa692834dd0d415a727c8"
Oct 31 13:21:22.416438 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.416418    5787 scope.go:122] "RemoveContainer" containerID="2a1cf4987942614247b1a2f89cb9d6f19398db2cf1aef6cf8dda9265059b77fe"
Oct 31 13:21:22.424930 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.424909    5787 scope.go:122] "RemoveContainer" containerID="d862c33421da9249aa5cfff45c509d611f59e9a812d41d34473f2a10d5161fa9"
Oct 31 13:21:22.434196 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.434167    5787 scope.go:122] "RemoveContainer" containerID="fcde411c9a378442af3626a800095f2c9e69f260bd1551ef7fbcde30f307c6cf"
Oct 31 13:21:22.445595 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.445566    5787 scope.go:122] "RemoveContainer" containerID="ab64b75a219a1932f280078f8f83f3cdaf9542789461ba0f6b3d16f90cb1f515"
Oct 31 13:21:22.454213 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.454191    5787 scope.go:122] "RemoveContainer" containerID="2a798ebd143819ed2526e276e4fa130076ac50d932c97636990ce98a2568b3aa"
Oct 31 13:21:22.463079 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:22.463053    5787 scope.go:122] "RemoveContainer" containerID="506c96ca7c9f4a6cf4040ebd241d7af03566694d2c8d28e8d7f293c03f2a4729"
Oct 31 13:21:25.119490 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.119454    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"]
Oct 31 13:21:25.119936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.119897    5787 cpu_manager.go:414] "containerMap: removing container" podUID="ede0ed7e-5c00-45af-b071-6bd7984a100c" containerName="revoke-keys"
Oct 31 13:21:25.119936 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.119911    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede0ed7e-5c00-45af-b071-6bd7984a100c" containerName="revoke-keys"
Oct 31 13:21:25.120029 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.119991    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede0ed7e-5c00-45af-b071-6bd7984a100c" containerName="revoke-keys"
Oct 31 13:21:25.126857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.126831    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.131837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.131540    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:21:25.131837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.131605    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:21:25.132078 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.131924    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:21:25.132164 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.132137    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:21:25.132262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.132234    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"]
Oct 31 13:21:25.245995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.245955    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-blcjx\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.246201 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.246007    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wwtd\" (UniqueName: \"kubernetes.io/projected/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-kube-api-access-5wwtd\") pod \"maas-api-revoke-sub-5a29efb3-blcjx\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.246201 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.246031    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-blcjx\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.347795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.347753    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-blcjx\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.347991 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.347804    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wwtd\" (UniqueName: \"kubernetes.io/projected/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-kube-api-access-5wwtd\") pod \"maas-api-revoke-sub-5a29efb3-blcjx\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.347991 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.347940    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-blcjx\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.348513 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.348490    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-blcjx\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.350405 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.350376    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-blcjx\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.361173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.361148    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wwtd\" (UniqueName: \"kubernetes.io/projected/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-kube-api-access-5wwtd\") pod \"maas-api-revoke-sub-5a29efb3-blcjx\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.443283 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.443242    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:25.723671 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:25.723646    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"]
Oct 31 13:21:25.725983 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:21:25.725951    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffcf2740_24c5_4fc6_94a6_fd4e8cc9771b.slice/crio-7d006d6b5fed701805ff9c0c4957ba0be15958ea9281bfd80da2954777bc7ce2 WatchSource:0}: Error finding container 7d006d6b5fed701805ff9c0c4957ba0be15958ea9281bfd80da2954777bc7ce2: Status 404 returned error can't find the container with id 7d006d6b5fed701805ff9c0c4957ba0be15958ea9281bfd80da2954777bc7ce2
Oct 31 13:21:26.262377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:26.262333    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx" event={"ID":"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b","Type":"ContainerStarted","Data":"226890a5729da30bef99730117a54283b8a9ad9c477e552c7d9c498b624063dd"}
Oct 31 13:21:26.262914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:26.262399    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx" event={"ID":"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b","Type":"ContainerStarted","Data":"7d006d6b5fed701805ff9c0c4957ba0be15958ea9281bfd80da2954777bc7ce2"}
Oct 31 13:21:26.284803 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:26.284737    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx" podStartSLOduration=1.284718346 podStartE2EDuration="1.284718346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:26.283177681 +0000 UTC m=+1265.699119522" watchObservedRunningTime="2026-10-31 13:21:26.284718346 +0000 UTC m=+1265.700660180"
Oct 31 13:21:27.296729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:27.295841    5787 generic.go:356] "Generic (PLEG): container finished" podID="ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b" containerID="226890a5729da30bef99730117a54283b8a9ad9c477e552c7d9c498b624063dd" exitCode=0
Oct 31 13:21:27.296729 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:27.296055    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx" event={"ID":"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b","Type":"ContainerDied","Data":"226890a5729da30bef99730117a54283b8a9ad9c477e552c7d9c498b624063dd"}
Oct 31 13:21:28.630068 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.630034    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:28.717943 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.717913    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-xks-ca\") pod \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") "
Oct 31 13:21:28.718108 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.718004    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-kube-api-access-5wwtd\" (UniqueName: \"kubernetes.io/projected/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-kube-api-access-5wwtd\") pod \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") "
Oct 31 13:21:28.718169 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.718109    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-openshift-ca\") pod \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\" (UID: \"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b\") "
Oct 31 13:21:28.718464 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.718439    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-openshift-ca" pod "ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b" (UID: "ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:21:28.720942 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.720911    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-xks-ca" pod "ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b" (UID: "ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:21:28.720942 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.720924    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-kube-api-access-5wwtd" pod "ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b" (UID: "ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b"). InnerVolumeSpecName "kube-api-access-5wwtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:21:28.819081 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.819043    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wwtd\" (UniqueName: \"kubernetes.io/projected/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-kube-api-access-5wwtd\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:28.819081 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.819074    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:28.819081 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:28.819088    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:21:29.313801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:29.313767    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"
Oct 31 13:21:29.313962 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:29.313795    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx" event={"ID":"ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b","Type":"ContainerDied","Data":"7d006d6b5fed701805ff9c0c4957ba0be15958ea9281bfd80da2954777bc7ce2"}
Oct 31 13:21:29.313962 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:29.313845    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d006d6b5fed701805ff9c0c4957ba0be15958ea9281bfd80da2954777bc7ce2"
Oct 31 13:21:40.851694 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:40.851656    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"]
Oct 31 13:21:40.856589 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:40.856565    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-kjtx9"]
Oct 31 13:21:42.704498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:42.704453    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a98eec-6227-4e17-af10-c313dd77f884" path="/var/lib/kubelet/pods/57a98eec-6227-4e17-af10-c313dd77f884/volumes"
Oct 31 13:21:48.246615 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:21:48.246580    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:21:55.024188 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:55.024146    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"]
Oct 31 13:21:55.027624 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:55.027597    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-tmczb"]
Oct 31 13:21:56.704896 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:21:56.704860    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64bf472f-5874-4a45-806c-49bd62934c1e" path="/var/lib/kubelet/pods/64bf472f-5874-4a45-806c-49bd62934c1e/volumes"
Oct 31 13:22:18.245071 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:22:18.245031    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:22:22.541588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.541553    5787 scope.go:122] "RemoveContainer" containerID="f480bfba1603856c166411cfca2aec8783a95a00adfc6c300f376a45a8aef507"
Oct 31 13:22:22.552126 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.552103    5787 scope.go:122] "RemoveContainer" containerID="4e570ab7319db24476faa4096ea332945acb0e887e6ffaba0b71d5a7399ecf5c"
Oct 31 13:22:22.560975 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.560954    5787 scope.go:122] "RemoveContainer" containerID="1abeea7fcaa808964861d2d09b1c465e8a408ebb17d0efa612c1e8b765345f56"
Oct 31 13:22:22.570432 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.570410    5787 scope.go:122] "RemoveContainer" containerID="5038e3d66a70284e1fc4c718d8fa9e474b2e6e935c95001d48a1dce8c0c9c184"
Oct 31 13:22:22.579143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.579121    5787 scope.go:122] "RemoveContainer" containerID="e5b8e967fcb8846e69bddbfd2775c65b2ebce65eff397e478e1ac5cd120f0b75"
Oct 31 13:22:22.588508 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.588489    5787 scope.go:122] "RemoveContainer" containerID="d2af4d61a4bd1ecbaa7cd5d8a3a70c9ec848bbcba3467d5184eda1a221a1405f"
Oct 31 13:22:22.596905 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.596882    5787 scope.go:122] "RemoveContainer" containerID="c71e02e55ef5d9c59e3d8ef1f96b212278afe2ab0412e599eb4308a89c37577d"
Oct 31 13:22:22.605583 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.605554    5787 scope.go:122] "RemoveContainer" containerID="9a5b64cfe7e7b7c981838fde56fed4e3f54f4ed5f2bcede31ab3831a11ce437c"
Oct 31 13:22:22.613891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.613870    5787 scope.go:122] "RemoveContainer" containerID="b4a35a8fe39d7dd5e5ffdc69421224c09005891f8c68c80fd4672bb88f1c2b30"
Oct 31 13:22:22.623305 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.623281    5787 scope.go:122] "RemoveContainer" containerID="3ba7d8cef9905de05d45f737763c34ce1d2aed1fda6d289386608fcf607e7cd2"
Oct 31 13:22:22.633930 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.633905    5787 scope.go:122] "RemoveContainer" containerID="ac68659275efb474db7b100d8b777aa60195f873d5c4435a7279172ad00ee68a"
Oct 31 13:22:22.644641 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.644618    5787 scope.go:122] "RemoveContainer" containerID="71ec49ffae3f6077521d7f3552cb36617b3e01ad27a3de6fee1c137271845f1f"
Oct 31 13:22:22.653110 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.653087    5787 scope.go:122] "RemoveContainer" containerID="be9156e4c43e94d437f9134b57e2b23bcad7864e49ca85fb42c7076db2107e4a"
Oct 31 13:22:22.661468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.661447    5787 scope.go:122] "RemoveContainer" containerID="c59e61d90bc230ed7980a792d100165f71c1a9b4fd1e6a81929686a514743d37"
Oct 31 13:22:22.670086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.670065    5787 scope.go:122] "RemoveContainer" containerID="3f85f7245083b90400f1e2d460a539419fc38b13709b7b9738e9770345e28536"
Oct 31 13:22:22.680510 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:22.680489    5787 scope.go:122] "RemoveContainer" containerID="4d08c400cc17ccda0a1c4112ea308734f990051cdd3d636fe0b24257c71f157b"
Oct 31 13:22:23.362455 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.362421    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"]
Oct 31 13:22:23.362912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.362892    5787 cpu_manager.go:414] "containerMap: removing container" podUID="ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b" containerName="revoke-keys"
Oct 31 13:22:23.362912 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.362910    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b" containerName="revoke-keys"
Oct 31 13:22:23.363102 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.363025    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b" containerName="revoke-keys"
Oct 31 13:22:23.368821 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.368779    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.378646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.378605    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-25d309\"/\"simulator-single-main-25d309-kserve-self-signed-certs\""
Oct 31 13:22:23.378828 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.378729    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-25d309\"/\"openshift-service-ca.crt\""
Oct 31 13:22:23.378896 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.378853    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-25d309\"/\"default-dockercfg-nvch4\""
Oct 31 13:22:23.378950 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.378931    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-25d309\"/\"kube-root-ca.crt\""
Oct 31 13:22:23.379055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.379041    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"]
Oct 31 13:22:23.496224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.496186    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-tmp-dir\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.496224 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.496228    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aade94f4-8105-4c72-ac7e-0b42406a743c-tls-certs\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.496458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.496313    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-dshm\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.496458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.496361    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-home\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.496458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.496401    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n82sp\" (UniqueName: \"kubernetes.io/projected/aade94f4-8105-4c72-ac7e-0b42406a743c-kube-api-access-n82sp\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.496458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.496425    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-model-cache\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.597632 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.597593    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-tmp-dir\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.598171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.597643    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aade94f4-8105-4c72-ac7e-0b42406a743c-tls-certs\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.598171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.597678    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-dshm\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.598171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.597732    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-home\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.598171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.597796    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n82sp\" (UniqueName: \"kubernetes.io/projected/aade94f4-8105-4c72-ac7e-0b42406a743c-kube-api-access-n82sp\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.598171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.597831    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-model-cache\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.598171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.598103    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-tmp-dir\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.598171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.598168    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-home\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.598477 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.598362    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-model-cache\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.600741 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.600694    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-dshm\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.600995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.600978    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aade94f4-8105-4c72-ac7e-0b42406a743c-tls-certs\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.611382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.611355    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n82sp\" (UniqueName: \"kubernetes.io/projected/aade94f4-8105-4c72-ac7e-0b42406a743c-kube-api-access-n82sp\") pod \"simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") " pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.683754 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.683655    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:23.993004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:23.992977    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"]
Oct 31 13:22:23.995847 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:22:23.995814    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaade94f4_8105_4c72_ac7e_0b42406a743c.slice/crio-fdca117a19ed75ccd51042aece09d8fdaed8378ef7b6d43e1bd36b7c14e5cc0f WatchSource:0}: Error finding container fdca117a19ed75ccd51042aece09d8fdaed8378ef7b6d43e1bd36b7c14e5cc0f: Status 404 returned error can't find the container with id fdca117a19ed75ccd51042aece09d8fdaed8378ef7b6d43e1bd36b7c14e5cc0f
Oct 31 13:22:24.706534 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:24.706496    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl" event={"ID":"aade94f4-8105-4c72-ac7e-0b42406a743c","Type":"ContainerStarted","Data":"2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663"}
Oct 31 13:22:24.707014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:24.706545    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl" event={"ID":"aade94f4-8105-4c72-ac7e-0b42406a743c","Type":"ContainerStarted","Data":"fdca117a19ed75ccd51042aece09d8fdaed8378ef7b6d43e1bd36b7c14e5cc0f"}
Oct 31 13:22:24.729770 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:24.729684    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl" podStartSLOduration=1.729667279 podStartE2EDuration="1.729667279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:22:24.728727158 +0000 UTC m=+1324.144669005" watchObservedRunningTime="2026-10-31 13:22:24.729667279 +0000 UTC m=+1324.145609117"
Oct 31 13:22:25.684523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:25.684491    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:33.683995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:33.683952    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:33.696867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:33.696839    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:33.772454 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:33.772426    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:22:48.028579 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:48.028542    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"]
Oct 31 13:22:48.031483 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:48.031458    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-gl8l6"]
Oct 31 13:22:48.245558 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:22:48.245517    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:22:48.704527 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:48.704492    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce4f8b7c-6cf4-44f0-aba7-9f48d798d425" path="/var/lib/kubelet/pods/ce4f8b7c-6cf4-44f0-aba7-9f48d798d425/volumes"
Oct 31 13:22:55.284004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.283969    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"]
Oct 31 13:22:55.287690 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.287673    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.292075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.292051    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:22:55.292961 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.292939    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:22:55.293102 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.293019    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:22:55.293102 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.293020    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:22:55.298996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.298973    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"]
Oct 31 13:22:55.321330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.321300    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b26686-zj4jf\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.321330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.321336    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b26686-zj4jf\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.321522 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.321362    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5brd\" (UniqueName: \"kubernetes.io/projected/4e9893a2-da1a-410a-85a7-14565155b107-kube-api-access-r5brd\") pod \"maas-api-revoke-sub-15b26686-zj4jf\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.424275 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.424233    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b26686-zj4jf\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.424451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.424294    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b26686-zj4jf\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.424451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.424337    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5brd\" (UniqueName: \"kubernetes.io/projected/4e9893a2-da1a-410a-85a7-14565155b107-kube-api-access-r5brd\") pod \"maas-api-revoke-sub-15b26686-zj4jf\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.425813 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.425679    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15b26686-zj4jf\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.429533 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.429503    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15b26686-zj4jf\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.437976 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.437947    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5brd\" (UniqueName: \"kubernetes.io/projected/4e9893a2-da1a-410a-85a7-14565155b107-kube-api-access-r5brd\") pod \"maas-api-revoke-sub-15b26686-zj4jf\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.602862 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.602761    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:55.881108 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.881080    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"]
Oct 31 13:22:55.883468 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:22:55.883436    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9893a2_da1a_410a_85a7_14565155b107.slice/crio-a7864a8f21d40cc65ebe8ded7e6afceee6097c6aac9ccec906f62c41ecc93272 WatchSource:0}: Error finding container a7864a8f21d40cc65ebe8ded7e6afceee6097c6aac9ccec906f62c41ecc93272: Status 404 returned error can't find the container with id a7864a8f21d40cc65ebe8ded7e6afceee6097c6aac9ccec906f62c41ecc93272
Oct 31 13:22:55.901580 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:55.901545    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf" event={"ID":"4e9893a2-da1a-410a-85a7-14565155b107","Type":"ContainerStarted","Data":"a7864a8f21d40cc65ebe8ded7e6afceee6097c6aac9ccec906f62c41ecc93272"}
Oct 31 13:22:56.908490 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:56.908396    5787 generic.go:356] "Generic (PLEG): container finished" podID="4e9893a2-da1a-410a-85a7-14565155b107" containerID="68c8dd5c2f734e9d2b11de47819ef330b0a736ec3660ec708f1f56d66e6ccf77" exitCode=0
Oct 31 13:22:56.908914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:56.908483    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf" event={"ID":"4e9893a2-da1a-410a-85a7-14565155b107","Type":"ContainerDied","Data":"68c8dd5c2f734e9d2b11de47819ef330b0a736ec3660ec708f1f56d66e6ccf77"}
Oct 31 13:22:58.057509 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.057478    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:58.152802 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.152768    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-xks-ca\") pod \"4e9893a2-da1a-410a-85a7-14565155b107\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") "
Oct 31 13:22:58.153014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.152828    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e9893a2-da1a-410a-85a7-14565155b107-kube-api-access-r5brd\" (UniqueName: \"kubernetes.io/projected/4e9893a2-da1a-410a-85a7-14565155b107-kube-api-access-r5brd\") pod \"4e9893a2-da1a-410a-85a7-14565155b107\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") "
Oct 31 13:22:58.153014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.152920    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-openshift-ca\") pod \"4e9893a2-da1a-410a-85a7-14565155b107\" (UID: \"4e9893a2-da1a-410a-85a7-14565155b107\") "
Oct 31 13:22:58.153290 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.153260    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-openshift-ca" pod "4e9893a2-da1a-410a-85a7-14565155b107" (UID: "4e9893a2-da1a-410a-85a7-14565155b107"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:22:58.155153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.155126    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9893a2-da1a-410a-85a7-14565155b107-kube-api-access-r5brd" pod "4e9893a2-da1a-410a-85a7-14565155b107" (UID: "4e9893a2-da1a-410a-85a7-14565155b107"). InnerVolumeSpecName "kube-api-access-r5brd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:22:58.155153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.155126    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-xks-ca" pod "4e9893a2-da1a-410a-85a7-14565155b107" (UID: "4e9893a2-da1a-410a-85a7-14565155b107"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:22:58.254158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.254124    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5brd\" (UniqueName: \"kubernetes.io/projected/4e9893a2-da1a-410a-85a7-14565155b107-kube-api-access-r5brd\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:58.254158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.254154    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:58.254158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.254165    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e9893a2-da1a-410a-85a7-14565155b107-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:22:58.920833 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.920794    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"
Oct 31 13:22:58.920833 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.920814    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf" event={"ID":"4e9893a2-da1a-410a-85a7-14565155b107","Type":"ContainerDied","Data":"a7864a8f21d40cc65ebe8ded7e6afceee6097c6aac9ccec906f62c41ecc93272"}
Oct 31 13:22:58.921050 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:22:58.920866    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7864a8f21d40cc65ebe8ded7e6afceee6097c6aac9ccec906f62c41ecc93272"
Oct 31 13:23:06.423901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.423811    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"]
Oct 31 13:23:06.424454 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.424290    5787 cpu_manager.go:414] "containerMap: removing container" podUID="4e9893a2-da1a-410a-85a7-14565155b107" containerName="revoke-keys"
Oct 31 13:23:06.424454 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.424308    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e9893a2-da1a-410a-85a7-14565155b107" containerName="revoke-keys"
Oct 31 13:23:06.424454 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.424411    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e9893a2-da1a-410a-85a7-14565155b107" containerName="revoke-keys"
Oct 31 13:23:06.427713 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.427684    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.436170 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.436140    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:23:06.436366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.436168    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:23:06.436465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.436189    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:23:06.436548 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.436273    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:23:06.437265 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.437240    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"]
Oct 31 13:23:06.539253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.539162    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxn9p\" (UniqueName: \"kubernetes.io/projected/f5225548-49e8-4e04-98e7-10204ea2545b-kube-api-access-jxn9p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.539253 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.539257    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.539485 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.539338    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.640442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.640400    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxn9p\" (UniqueName: \"kubernetes.io/projected/f5225548-49e8-4e04-98e7-10204ea2545b-kube-api-access-jxn9p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.640442 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.640441    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.640755 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.640505    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.641272 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.641244    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.643112 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.643092    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.654066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.654042    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxn9p\" (UniqueName: \"kubernetes.io/projected/f5225548-49e8-4e04-98e7-10204ea2545b-kube-api-access-jxn9p\") pod \"maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:06.741157 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:06.741108    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:07.024470 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:07.024438    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"]
Oct 31 13:23:07.026682 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:23:07.026657    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5225548_49e8_4e04_98e7_10204ea2545b.slice/crio-2d03f400f19ecc9b461123a349729a1f7750567387c584c26d9ae99a6d8b19f8 WatchSource:0}: Error finding container 2d03f400f19ecc9b461123a349729a1f7750567387c584c26d9ae99a6d8b19f8: Status 404 returned error can't find the container with id 2d03f400f19ecc9b461123a349729a1f7750567387c584c26d9ae99a6d8b19f8
Oct 31 13:23:07.967072 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:07.967040    5787 generic.go:356] "Generic (PLEG): container finished" podID="f5225548-49e8-4e04-98e7-10204ea2545b" containerID="3ac4dd607f66b96483993e90d85646aa86473feed2321ee927577357cb76b564" exitCode=0
Oct 31 13:23:07.967515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:07.967143    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f" event={"ID":"f5225548-49e8-4e04-98e7-10204ea2545b","Type":"ContainerDied","Data":"3ac4dd607f66b96483993e90d85646aa86473feed2321ee927577357cb76b564"}
Oct 31 13:23:07.967515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:07.967206    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f" event={"ID":"f5225548-49e8-4e04-98e7-10204ea2545b","Type":"ContainerStarted","Data":"2d03f400f19ecc9b461123a349729a1f7750567387c584c26d9ae99a6d8b19f8"}
Oct 31 13:23:09.026202 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.026163    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"]
Oct 31 13:23:09.029457 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.029307    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-wqgkw"]
Oct 31 13:23:09.120922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.120894    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:09.266255 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.266170    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-xks-ca\") pod \"f5225548-49e8-4e04-98e7-10204ea2545b\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") "
Oct 31 13:23:09.266255 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.266218    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5225548-49e8-4e04-98e7-10204ea2545b-kube-api-access-jxn9p\" (UniqueName: \"kubernetes.io/projected/f5225548-49e8-4e04-98e7-10204ea2545b-kube-api-access-jxn9p\") pod \"f5225548-49e8-4e04-98e7-10204ea2545b\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") "
Oct 31 13:23:09.266501 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.266351    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-openshift-ca\") pod \"f5225548-49e8-4e04-98e7-10204ea2545b\" (UID: \"f5225548-49e8-4e04-98e7-10204ea2545b\") "
Oct 31 13:23:09.266720 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.266677    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-openshift-ca" pod "f5225548-49e8-4e04-98e7-10204ea2545b" (UID: "f5225548-49e8-4e04-98e7-10204ea2545b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:23:09.268441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.268418    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-xks-ca" pod "f5225548-49e8-4e04-98e7-10204ea2545b" (UID: "f5225548-49e8-4e04-98e7-10204ea2545b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:23:09.268441 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.268421    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5225548-49e8-4e04-98e7-10204ea2545b-kube-api-access-jxn9p" pod "f5225548-49e8-4e04-98e7-10204ea2545b" (UID: "f5225548-49e8-4e04-98e7-10204ea2545b"). InnerVolumeSpecName "kube-api-access-jxn9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:23:09.367588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.367542    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:09.367588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.367581    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxn9p\" (UniqueName: \"kubernetes.io/projected/f5225548-49e8-4e04-98e7-10204ea2545b-kube-api-access-jxn9p\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:09.367588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.367591    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5225548-49e8-4e04-98e7-10204ea2545b-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:09.978595 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.978564    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"
Oct 31 13:23:09.978820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.978559    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f" event={"ID":"f5225548-49e8-4e04-98e7-10204ea2545b","Type":"ContainerDied","Data":"2d03f400f19ecc9b461123a349729a1f7750567387c584c26d9ae99a6d8b19f8"}
Oct 31 13:23:09.978820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:09.978689    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d03f400f19ecc9b461123a349729a1f7750567387c584c26d9ae99a6d8b19f8"
Oct 31 13:23:10.706220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:10.706188    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="414013a4-02e7-4b4d-8596-966e5383cb53" path="/var/lib/kubelet/pods/414013a4-02e7-4b4d-8596-966e5383cb53/volumes"
Oct 31 13:23:11.223282 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:11.223241    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"]
Oct 31 13:23:18.145226 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:18.145174    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"]
Oct 31 13:23:18.148059 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:18.148029    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-25d309-lbp6f"]
Oct 31 13:23:18.245570 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:23:18.245537    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:23:18.703796 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:18.703762    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5225548-49e8-4e04-98e7-10204ea2545b" path="/var/lib/kubelet/pods/f5225548-49e8-4e04-98e7-10204ea2545b/volumes"
Oct 31 13:23:20.039777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:20.039741    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"]
Oct 31 13:23:20.050388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:20.050351    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-hwxmh"]
Oct 31 13:23:20.706160 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:20.706133    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da40fb8-0a34-4f04-87bf-86e4d3559391" path="/var/lib/kubelet/pods/6da40fb8-0a34-4f04-87bf-86e4d3559391/volumes"
Oct 31 13:23:22.765719 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:22.765674    5787 scope.go:122] "RemoveContainer" containerID="cfe4e242df9877f1578416f9869151a544eee9b7f114664c329d843e28b7f2d8"
Oct 31 13:23:26.244889 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.244813    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl" podUID="aade94f4-8105-4c72-ac7e-0b42406a743c" containerName="main" containerID="cri-o://2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663" gracePeriod=45
Oct 31 13:23:26.494673 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.494647    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:23:26.541910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.541825    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-tmp-dir\") pod \"aade94f4-8105-4c72-ac7e-0b42406a743c\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") "
Oct 31 13:23:26.541910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.541874    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aade94f4-8105-4c72-ac7e-0b42406a743c-kube-api-access-n82sp\" (UniqueName: \"kubernetes.io/projected/aade94f4-8105-4c72-ac7e-0b42406a743c-kube-api-access-n82sp\") pod \"aade94f4-8105-4c72-ac7e-0b42406a743c\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") "
Oct 31 13:23:26.542143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.541937    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-model-cache\") pod \"aade94f4-8105-4c72-ac7e-0b42406a743c\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") "
Oct 31 13:23:26.542143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.541974    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-dshm\") pod \"aade94f4-8105-4c72-ac7e-0b42406a743c\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") "
Oct 31 13:23:26.542143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.542001    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aade94f4-8105-4c72-ac7e-0b42406a743c-tls-certs\" (UniqueName: \"kubernetes.io/secret/aade94f4-8105-4c72-ac7e-0b42406a743c-tls-certs\") pod \"aade94f4-8105-4c72-ac7e-0b42406a743c\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") "
Oct 31 13:23:26.542143 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.542049    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-home\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-home\") pod \"aade94f4-8105-4c72-ac7e-0b42406a743c\" (UID: \"aade94f4-8105-4c72-ac7e-0b42406a743c\") "
Oct 31 13:23:26.542366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.542140    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-tmp-dir" pod "aade94f4-8105-4c72-ac7e-0b42406a743c" (UID: "aade94f4-8105-4c72-ac7e-0b42406a743c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:23:26.542366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.542196    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-model-cache" pod "aade94f4-8105-4c72-ac7e-0b42406a743c" (UID: "aade94f4-8105-4c72-ac7e-0b42406a743c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:23:26.542366 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.542348    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-home" pod "aade94f4-8105-4c72-ac7e-0b42406a743c" (UID: "aade94f4-8105-4c72-ac7e-0b42406a743c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:23:26.542544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.542435    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:26.542544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.542455    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:26.542544 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.542467    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:26.544553 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.544517    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aade94f4-8105-4c72-ac7e-0b42406a743c-kube-api-access-n82sp" pod "aade94f4-8105-4c72-ac7e-0b42406a743c" (UID: "aade94f4-8105-4c72-ac7e-0b42406a743c"). InnerVolumeSpecName "kube-api-access-n82sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:23:26.544662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.544616    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aade94f4-8105-4c72-ac7e-0b42406a743c-tls-certs" pod "aade94f4-8105-4c72-ac7e-0b42406a743c" (UID: "aade94f4-8105-4c72-ac7e-0b42406a743c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:23:26.544798 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.544779    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-dshm" pod "aade94f4-8105-4c72-ac7e-0b42406a743c" (UID: "aade94f4-8105-4c72-ac7e-0b42406a743c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:23:26.643637 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.643602    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n82sp\" (UniqueName: \"kubernetes.io/projected/aade94f4-8105-4c72-ac7e-0b42406a743c-kube-api-access-n82sp\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:26.643637 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.643631    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aade94f4-8105-4c72-ac7e-0b42406a743c-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:26.643882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:26.643645    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aade94f4-8105-4c72-ac7e-0b42406a743c-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:23:27.068990 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:27.068956    5787 generic.go:356] "Generic (PLEG): container finished" podID="aade94f4-8105-4c72-ac7e-0b42406a743c" containerID="2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663" exitCode=0
Oct 31 13:23:27.069167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:27.069000    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"
Oct 31 13:23:27.069167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:27.069043    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl" event={"ID":"aade94f4-8105-4c72-ac7e-0b42406a743c","Type":"ContainerDied","Data":"2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663"}
Oct 31 13:23:27.069167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:27.069084    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl" event={"ID":"aade94f4-8105-4c72-ac7e-0b42406a743c","Type":"ContainerDied","Data":"fdca117a19ed75ccd51042aece09d8fdaed8378ef7b6d43e1bd36b7c14e5cc0f"}
Oct 31 13:23:27.069167 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:27.069102    5787 scope.go:122] "RemoveContainer" containerID="2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663"
Oct 31 13:23:27.081536 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:27.081519    5787 scope.go:122] "RemoveContainer" containerID="2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663"
Oct 31 13:23:27.081893 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:23:27.081875    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663\": container with ID starting with 2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663 not found: ID does not exist" containerID="2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663"
Oct 31 13:23:27.081985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:27.081901    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663"} err="failed to get container status \"2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663\": rpc error: code = NotFound desc = could not find container \"2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663\": container with ID starting with 2e06d704537810240aca1608c4f44454f3099abe604f7516d54bf8c700aec663 not found: ID does not exist"
Oct 31 13:23:27.091462 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:27.091436    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"]
Oct 31 13:23:27.097957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:27.097930    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-25d309/simulator-single-main-25d309-kserve-6f5f9cfdb5-pnnvl"]
Oct 31 13:23:28.704119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:28.704083    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aade94f4-8105-4c72-ac7e-0b42406a743c" path="/var/lib/kubelet/pods/aade94f4-8105-4c72-ac7e-0b42406a743c/volumes"
Oct 31 13:23:47.025040 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:47.025005    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"]
Oct 31 13:23:47.028655 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:47.028629    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5ghr2"]
Oct 31 13:23:48.246589 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:23:48.246544    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:23:48.705131 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:48.705100    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3636f9e4-b2e7-466a-bddf-b756496fe3b9" path="/var/lib/kubelet/pods/3636f9e4-b2e7-466a-bddf-b756496fe3b9/volumes"
Oct 31 13:23:57.027669 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:57.027628    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"]
Oct 31 13:23:57.030627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:57.030590    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-ttmjx"]
Oct 31 13:23:58.704937 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:23:58.704904    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63cb9375-5f78-4f12-906e-9f6e6b274a96" path="/var/lib/kubelet/pods/63cb9375-5f78-4f12-906e-9f6e6b274a96/volumes"
Oct 31 13:24:18.028324 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:18.028276    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"]
Oct 31 13:24:18.036299 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:18.036262    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-5pqh2"]
Oct 31 13:24:18.245959 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:24:18.245917    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:24:18.704631 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:18.704598    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d10be3-9a59-4528-9f8a-1fc29d912e1d" path="/var/lib/kubelet/pods/07d10be3-9a59-4528-9f8a-1fc29d912e1d/volumes"
Oct 31 13:24:22.788008 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:22.787957    5787 scope.go:122] "RemoveContainer" containerID="5926d96fc945961b46f2b5a0508a4ea3b6a0cfb133521fb3f410c9174f8ddfcc"
Oct 31 13:24:22.796687 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:22.796661    5787 scope.go:122] "RemoveContainer" containerID="fcee3478b563ab83a655fa457ad6ca78562be641833411a45c37756e1a092994"
Oct 31 13:24:22.805174 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:22.805154    5787 scope.go:122] "RemoveContainer" containerID="15950c801495e20c0dc95d0bd080462b430892128f88c823ef76840dbb29394e"
Oct 31 13:24:36.040162 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:36.040123    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"]
Oct 31 13:24:36.043446 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:36.043417    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-sv4m6"]
Oct 31 13:24:36.705617 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:36.705578    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17" path="/var/lib/kubelet/pods/8c57ffcb-8fd1-4ce1-a845-b83e4a5efe17/volumes"
Oct 31 13:24:48.246394 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:24:48.246355    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:24:56.032940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:56.032904    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"]
Oct 31 13:24:56.036330 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:56.036306    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-vj2bg"]
Oct 31 13:24:56.704175 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:24:56.704139    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65dc6323-857b-40bf-946c-a8edf854a44e" path="/var/lib/kubelet/pods/65dc6323-857b-40bf-946c-a8edf854a44e/volumes"
Oct 31 13:25:14.026161 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:25:14.026124    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"]
Oct 31 13:25:14.033251 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:25:14.033204    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rklgg"]
Oct 31 13:25:14.705683 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:25:14.705645    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af2e4541-14bb-4453-9429-441c6ae06326" path="/var/lib/kubelet/pods/af2e4541-14bb-4453-9429-441c6ae06326/volumes"
Oct 31 13:25:18.246247 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:25:18.246211    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:25:22.837899 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:25:22.837866    5787 scope.go:122] "RemoveContainer" containerID="3230c3beed5d9b1d888a2b2a647597a4dc88e8ec69349c91fc59d8290b09207f"
Oct 31 13:25:22.846980 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:25:22.846958    5787 scope.go:122] "RemoveContainer" containerID="8764eae399dfd25b52633ef4c1d83066a7a799dfcf18640dc1a44a5c865090b3"
Oct 31 13:25:22.855464 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:25:22.855441    5787 scope.go:122] "RemoveContainer" containerID="f776201fcd19cbc137a0269320219eecc6ba90aa0c218bad3b0e684b8a78f215"
Oct 31 13:25:48.245666 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:25:48.245624    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:26:00.710424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.710337    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"]
Oct 31 13:26:00.712928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.710788    5787 cpu_manager.go:414] "containerMap: removing container" podUID="aade94f4-8105-4c72-ac7e-0b42406a743c" containerName="main"
Oct 31 13:26:00.712928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.710802    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aade94f4-8105-4c72-ac7e-0b42406a743c" containerName="main"
Oct 31 13:26:00.712928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.710844    5787 cpu_manager.go:414] "containerMap: removing container" podUID="f5225548-49e8-4e04-98e7-10204ea2545b" containerName="revoke-keys"
Oct 31 13:26:00.712928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.710849    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5225548-49e8-4e04-98e7-10204ea2545b" containerName="revoke-keys"
Oct 31 13:26:00.712928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.710921    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5225548-49e8-4e04-98e7-10204ea2545b" containerName="revoke-keys"
Oct 31 13:26:00.712928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.710933    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="aade94f4-8105-4c72-ac7e-0b42406a743c" containerName="main"
Oct 31 13:26:00.714423 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.714404    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.719932 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.719910    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2c64dd\"/\"default-dockercfg-xtt5q\""
Oct 31 13:26:00.720086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.719912    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2c64dd\"/\"openshift-service-ca.crt\""
Oct 31 13:26:00.720170 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.719918    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-2c64dd\"/\"kube-root-ca.crt\""
Oct 31 13:26:00.720170 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.719920    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-2c64dd\"/\"simulator-single-main-2c64dd-kserve-self-signed-certs\""
Oct 31 13:26:00.725288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.725235    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"]
Oct 31 13:26:00.801892 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.801851    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-model-cache\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.801892 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.801893    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tls-certs\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.802197 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.802173    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fstpx\" (UniqueName: \"kubernetes.io/projected/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-kube-api-access-fstpx\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.802262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.802247    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-home\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.802492 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.802465    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tmp-dir\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.802744 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.802722    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-dshm\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.903762 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.903687    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tmp-dir\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.903967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.903809    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-dshm\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.903967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.903877    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-model-cache\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.903967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.903903    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tls-certs\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.903967 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.903957    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fstpx\" (UniqueName: \"kubernetes.io/projected/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-kube-api-access-fstpx\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.904195 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.903987    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-home\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.904446 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.904405    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-home\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.904582 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.904485    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tmp-dir\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.904775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.904752    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-model-cache\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.909547 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.909517    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-dshm\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.912245 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.911529    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tls-certs\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:00.920026 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:00.919995    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fstpx\" (UniqueName: \"kubernetes.io/projected/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-kube-api-access-fstpx\") pod \"simulator-single-main-2c64dd-kserve-77ff975b99-sbvng\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") " pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:01.029458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:01.029364    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:01.312608 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:01.312582    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"]
Oct 31 13:26:01.314727 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:26:01.314685    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda799f8c8_7b22_4d82_a6ec_f32e9e63d14b.slice/crio-763fb81db64c40ca366f80a85ca4ce552cbe7974b9bf72937aa1b4c307485dba WatchSource:0}: Error finding container 763fb81db64c40ca366f80a85ca4ce552cbe7974b9bf72937aa1b4c307485dba: Status 404 returned error can't find the container with id 763fb81db64c40ca366f80a85ca4ce552cbe7974b9bf72937aa1b4c307485dba
Oct 31 13:26:01.871814 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:01.871771    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng" event={"ID":"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b","Type":"ContainerStarted","Data":"b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea"}
Oct 31 13:26:01.872218 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:01.871825    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng" event={"ID":"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b","Type":"ContainerStarted","Data":"763fb81db64c40ca366f80a85ca4ce552cbe7974b9bf72937aa1b4c307485dba"}
Oct 31 13:26:01.905264 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:01.905196    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng" podStartSLOduration=1.905180106 podStartE2EDuration="1.905180106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:01.903980219 +0000 UTC m=+1541.319922058" watchObservedRunningTime="2026-10-31 13:26:01.905180106 +0000 UTC m=+1541.321121939"
Oct 31 13:26:02.040525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:02.040489    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:11.030137 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:11.030098    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:11.042887 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:11.042859    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:11.933809 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:11.933776    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:26:18.245012 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:26:18.244971    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:26:19.025185 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:19.025145    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"]
Oct 31 13:26:19.028571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:19.028542    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-k6zz6"]
Oct 31 13:26:20.705181 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:20.705148    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede0ed7e-5c00-45af-b071-6bd7984a100c" path="/var/lib/kubelet/pods/ede0ed7e-5c00-45af-b071-6bd7984a100c/volumes"
Oct 31 13:26:22.883811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:22.883776    5787 scope.go:122] "RemoveContainer" containerID="26b306e62956aaa0bfca2a60c489ffdabb1e2ac8362dd8b2bb13d95f6ab7fac1"
Oct 31 13:26:22.892316 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:22.892294    5787 scope.go:122] "RemoveContainer" containerID="f8d6a304931c915b5749ad00af5cbf8e15b78ddbc7a40c73067c81fc7399a7d0"
Oct 31 13:26:22.900339 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:22.900318    5787 scope.go:122] "RemoveContainer" containerID="1460ee20aeca4731ecf658588a258a935710513c698313bb9e76aee61c293d6c"
Oct 31 13:26:22.908032 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:22.908013    5787 scope.go:122] "RemoveContainer" containerID="8706ed8520e40ef2965746e8ee73ed2f4a4eb7b7e286e028d421912322d68b37"
Oct 31 13:26:29.025142 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.025100    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"]
Oct 31 13:26:29.028333 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.028305    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-blcjx"]
Oct 31 13:26:29.253452 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.253411    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"]
Oct 31 13:26:29.258216 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.258194    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.266876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.266851    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:26:29.266876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.266849    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:26:29.267112 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.266907    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:26:29.267112 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.266914    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:26:29.268946 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.268923    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"]
Oct 31 13:26:29.402577 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.402494    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4ab5cf0-srswb\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.402577 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.402554    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdpbj\" (UniqueName: \"kubernetes.io/projected/d5a3b647-83c4-4e73-8460-26eb07319a62-kube-api-access-zdpbj\") pod \"maas-api-revoke-sub-e4ab5cf0-srswb\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.402800 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.402656    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4ab5cf0-srswb\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.503273 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.503240    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4ab5cf0-srswb\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.503458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.503317    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdpbj\" (UniqueName: \"kubernetes.io/projected/d5a3b647-83c4-4e73-8460-26eb07319a62-kube-api-access-zdpbj\") pod \"maas-api-revoke-sub-e4ab5cf0-srswb\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.503458 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.503357    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4ab5cf0-srswb\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.503913 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.503890    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4ab5cf0-srswb\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.505801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.505780    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4ab5cf0-srswb\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.518985 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.518955    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdpbj\" (UniqueName: \"kubernetes.io/projected/d5a3b647-83c4-4e73-8460-26eb07319a62-kube-api-access-zdpbj\") pod \"maas-api-revoke-sub-e4ab5cf0-srswb\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.572015 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.571983    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:29.850031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:29.850002    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"]
Oct 31 13:26:29.852748 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:26:29.852695    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a3b647_83c4_4e73_8460_26eb07319a62.slice/crio-f9d96d0f4eab973c14ef63a2c22a2753686d4f8ea8512a7f9f0775ec9a32c0f8 WatchSource:0}: Error finding container f9d96d0f4eab973c14ef63a2c22a2753686d4f8ea8512a7f9f0775ec9a32c0f8: Status 404 returned error can't find the container with id f9d96d0f4eab973c14ef63a2c22a2753686d4f8ea8512a7f9f0775ec9a32c0f8
Oct 31 13:26:30.023885 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:30.023850    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb" event={"ID":"d5a3b647-83c4-4e73-8460-26eb07319a62","Type":"ContainerStarted","Data":"f9d96d0f4eab973c14ef63a2c22a2753686d4f8ea8512a7f9f0775ec9a32c0f8"}
Oct 31 13:26:30.705193 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:30.705158    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b" path="/var/lib/kubelet/pods/ffcf2740-24c5-4fc6-94a6-fd4e8cc9771b/volumes"
Oct 31 13:26:31.031345 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:31.031311    5787 generic.go:356] "Generic (PLEG): container finished" podID="d5a3b647-83c4-4e73-8460-26eb07319a62" containerID="547ddd1ad7beb4c3a6ec3127aa976b48edd068af2368d1a952062786101446b7" exitCode=0
Oct 31 13:26:31.031521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:31.031394    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb" event={"ID":"d5a3b647-83c4-4e73-8460-26eb07319a62","Type":"ContainerDied","Data":"547ddd1ad7beb4c3a6ec3127aa976b48edd068af2368d1a952062786101446b7"}
Oct 31 13:26:32.172759 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.172729    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:32.340387 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.340301    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a3b647-83c4-4e73-8460-26eb07319a62-kube-api-access-zdpbj\" (UniqueName: \"kubernetes.io/projected/d5a3b647-83c4-4e73-8460-26eb07319a62-kube-api-access-zdpbj\") pod \"d5a3b647-83c4-4e73-8460-26eb07319a62\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") "
Oct 31 13:26:32.340543 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.340411    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-xks-ca\") pod \"d5a3b647-83c4-4e73-8460-26eb07319a62\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") "
Oct 31 13:26:32.340543 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.340460    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-openshift-ca\") pod \"d5a3b647-83c4-4e73-8460-26eb07319a62\" (UID: \"d5a3b647-83c4-4e73-8460-26eb07319a62\") "
Oct 31 13:26:32.340928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.340891    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-openshift-ca" pod "d5a3b647-83c4-4e73-8460-26eb07319a62" (UID: "d5a3b647-83c4-4e73-8460-26eb07319a62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:26:32.342621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.342594    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a3b647-83c4-4e73-8460-26eb07319a62-kube-api-access-zdpbj" pod "d5a3b647-83c4-4e73-8460-26eb07319a62" (UID: "d5a3b647-83c4-4e73-8460-26eb07319a62"). InnerVolumeSpecName "kube-api-access-zdpbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:26:32.342738 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.342595    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-xks-ca" pod "d5a3b647-83c4-4e73-8460-26eb07319a62" (UID: "d5a3b647-83c4-4e73-8460-26eb07319a62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:26:32.441426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.441385    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:32.441426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.441420    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3b647-83c4-4e73-8460-26eb07319a62-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:32.441426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:32.441431    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdpbj\" (UniqueName: \"kubernetes.io/projected/d5a3b647-83c4-4e73-8460-26eb07319a62-kube-api-access-zdpbj\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:33.047236 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:33.047150    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"
Oct 31 13:26:33.047389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:33.047149    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb" event={"ID":"d5a3b647-83c4-4e73-8460-26eb07319a62","Type":"ContainerDied","Data":"f9d96d0f4eab973c14ef63a2c22a2753686d4f8ea8512a7f9f0775ec9a32c0f8"}
Oct 31 13:26:33.047389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:33.047284    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9d96d0f4eab973c14ef63a2c22a2753686d4f8ea8512a7f9f0775ec9a32c0f8"
Oct 31 13:26:40.187577 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.187541    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"]
Oct 31 13:26:40.188023 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.187952    5787 cpu_manager.go:414] "containerMap: removing container" podUID="d5a3b647-83c4-4e73-8460-26eb07319a62" containerName="revoke-keys"
Oct 31 13:26:40.188023 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.187974    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a3b647-83c4-4e73-8460-26eb07319a62" containerName="revoke-keys"
Oct 31 13:26:40.188173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.188054    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a3b647-83c4-4e73-8460-26eb07319a62" containerName="revoke-keys"
Oct 31 13:26:40.191034 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.191018    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.195945 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.195927    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:26:40.197208 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.197192    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:26:40.197312 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.197294    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:26:40.197612 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.197324    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:26:40.200970 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.200948    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"]
Oct 31 13:26:40.328002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.327966    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2h9\" (UniqueName: \"kubernetes.io/projected/0ee08e21-0017-4597-86e5-d97ec5f8483b-kube-api-access-kr2h9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.328191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.328045    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.328191 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.328147    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.429317 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.429279    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2h9\" (UniqueName: \"kubernetes.io/projected/0ee08e21-0017-4597-86e5-d97ec5f8483b-kube-api-access-kr2h9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.429499 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.429361    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.429499 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.429413    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.430060 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.430034    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.431890 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.431869    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.442639 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.442582    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2h9\" (UniqueName: \"kubernetes.io/projected/0ee08e21-0017-4597-86e5-d97ec5f8483b-kube-api-access-kr2h9\") pod \"maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.504347 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.504321    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:40.779446 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:40.779424    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"]
Oct 31 13:26:40.782108 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:26:40.782075    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ee08e21_0017_4597_86e5_d97ec5f8483b.slice/crio-c93a0cf971c60a318b23281d0eb5edad1574bab032af9f426e3430c05ac77c8f WatchSource:0}: Error finding container c93a0cf971c60a318b23281d0eb5edad1574bab032af9f426e3430c05ac77c8f: Status 404 returned error can't find the container with id c93a0cf971c60a318b23281d0eb5edad1574bab032af9f426e3430c05ac77c8f
Oct 31 13:26:41.085496 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:41.085457    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g" event={"ID":"0ee08e21-0017-4597-86e5-d97ec5f8483b","Type":"ContainerStarted","Data":"41c92b8e7be0eb1658c4affe015bfafb7f67be4ec8f217d7d033608a7ba86807"}
Oct 31 13:26:41.085647 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:41.085514    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g" event={"ID":"0ee08e21-0017-4597-86e5-d97ec5f8483b","Type":"ContainerStarted","Data":"c93a0cf971c60a318b23281d0eb5edad1574bab032af9f426e3430c05ac77c8f"}
Oct 31 13:26:41.107814 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:41.107768    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g" podStartSLOduration=1.1077558889999999 podStartE2EDuration="1.107755889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:41.106245533 +0000 UTC m=+1580.522187385" watchObservedRunningTime="2026-10-31 13:26:41.107755889 +0000 UTC m=+1580.523697728"
Oct 31 13:26:42.091679 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:42.091641    5787 generic.go:356] "Generic (PLEG): container finished" podID="0ee08e21-0017-4597-86e5-d97ec5f8483b" containerID="41c92b8e7be0eb1658c4affe015bfafb7f67be4ec8f217d7d033608a7ba86807" exitCode=0
Oct 31 13:26:42.092117 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:42.091735    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g" event={"ID":"0ee08e21-0017-4597-86e5-d97ec5f8483b","Type":"ContainerDied","Data":"41c92b8e7be0eb1658c4affe015bfafb7f67be4ec8f217d7d033608a7ba86807"}
Oct 31 13:26:43.231918 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.231891    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:43.261075 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.261036    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-xks-ca\") pod \"0ee08e21-0017-4597-86e5-d97ec5f8483b\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") "
Oct 31 13:26:43.261250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.261087    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ee08e21-0017-4597-86e5-d97ec5f8483b-kube-api-access-kr2h9\" (UniqueName: \"kubernetes.io/projected/0ee08e21-0017-4597-86e5-d97ec5f8483b-kube-api-access-kr2h9\") pod \"0ee08e21-0017-4597-86e5-d97ec5f8483b\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") "
Oct 31 13:26:43.261250 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.261223    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-openshift-ca\") pod \"0ee08e21-0017-4597-86e5-d97ec5f8483b\" (UID: \"0ee08e21-0017-4597-86e5-d97ec5f8483b\") "
Oct 31 13:26:43.261589 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.261566    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-openshift-ca" pod "0ee08e21-0017-4597-86e5-d97ec5f8483b" (UID: "0ee08e21-0017-4597-86e5-d97ec5f8483b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:26:43.263256 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.263230    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-xks-ca" pod "0ee08e21-0017-4597-86e5-d97ec5f8483b" (UID: "0ee08e21-0017-4597-86e5-d97ec5f8483b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:26:43.263379 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.263301    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ee08e21-0017-4597-86e5-d97ec5f8483b-kube-api-access-kr2h9" pod "0ee08e21-0017-4597-86e5-d97ec5f8483b" (UID: "0ee08e21-0017-4597-86e5-d97ec5f8483b"). InnerVolumeSpecName "kube-api-access-kr2h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:26:43.362318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.362234    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:43.362318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.362265    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr2h9\" (UniqueName: \"kubernetes.io/projected/0ee08e21-0017-4597-86e5-d97ec5f8483b-kube-api-access-kr2h9\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:43.362318 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:43.362276    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ee08e21-0017-4597-86e5-d97ec5f8483b-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:26:44.106977 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:44.106949    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"
Oct 31 13:26:44.107166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:44.106946    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g" event={"ID":"0ee08e21-0017-4597-86e5-d97ec5f8483b","Type":"ContainerDied","Data":"c93a0cf971c60a318b23281d0eb5edad1574bab032af9f426e3430c05ac77c8f"}
Oct 31 13:26:44.107166 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:44.107092    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c93a0cf971c60a318b23281d0eb5edad1574bab032af9f426e3430c05ac77c8f"
Oct 31 13:26:45.051288 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:45.051252    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"]
Oct 31 13:26:48.245526 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:26:48.245481    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:26:51.753381 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:51.753341    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"]
Oct 31 13:26:51.758662 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:51.758627    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-2c64dd-7fm2g"]
Oct 31 13:26:52.704033 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:26:52.704000    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ee08e21-0017-4597-86e5-d97ec5f8483b" path="/var/lib/kubelet/pods/0ee08e21-0017-4597-86e5-d97ec5f8483b/volumes"
Oct 31 13:27:00.071606 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.071568    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng" podUID="a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" containerName="main" containerID="cri-o://b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea" gracePeriod=45
Oct 31 13:27:00.319059 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.319036    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:27:00.434573 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.434485    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tls-certs\" (UniqueName: \"kubernetes.io/secret/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tls-certs\") pod \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") "
Oct 31 13:27:00.434573 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.434565    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-home\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-home\") pod \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") "
Oct 31 13:27:00.434842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.434617    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-model-cache\") pod \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") "
Oct 31 13:27:00.434842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.434764    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tmp-dir\") pod \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") "
Oct 31 13:27:00.434842 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.434793    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-dshm\") pod \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") "
Oct 31 13:27:00.434995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.434844    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-kube-api-access-fstpx\" (UniqueName: \"kubernetes.io/projected/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-kube-api-access-fstpx\") pod \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\" (UID: \"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b\") "
Oct 31 13:27:00.434995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.434829    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-home" pod "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" (UID: "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:27:00.434995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.434910    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-model-cache" pod "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" (UID: "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:27:00.435124 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.435007    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tmp-dir" pod "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" (UID: "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:27:00.435177 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.435162    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:00.435221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.435180    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:00.435221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.435189    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:00.436903 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.436871    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-dshm" pod "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" (UID: "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:27:00.437008 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.436936    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tls-certs" pod "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" (UID: "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:27:00.437056 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.437034    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-kube-api-access-fstpx" pod "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" (UID: "a799f8c8-7b22-4d82-a6ec-f32e9e63d14b"). InnerVolumeSpecName "kube-api-access-fstpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:27:00.536049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.536018    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:00.536049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.536045    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fstpx\" (UniqueName: \"kubernetes.io/projected/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-kube-api-access-fstpx\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:00.536049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:00.536056    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:27:01.189028 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:01.188991    5787 generic.go:356] "Generic (PLEG): container finished" podID="a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" containerID="b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea" exitCode=0
Oct 31 13:27:01.189426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:01.189055    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"
Oct 31 13:27:01.189426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:01.189081    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng" event={"ID":"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b","Type":"ContainerDied","Data":"b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea"}
Oct 31 13:27:01.189426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:01.189124    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng" event={"ID":"a799f8c8-7b22-4d82-a6ec-f32e9e63d14b","Type":"ContainerDied","Data":"763fb81db64c40ca366f80a85ca4ce552cbe7974b9bf72937aa1b4c307485dba"}
Oct 31 13:27:01.189426 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:01.189143    5787 scope.go:122] "RemoveContainer" containerID="b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea"
Oct 31 13:27:01.197902 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:01.197875    5787 scope.go:122] "RemoveContainer" containerID="b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea"
Oct 31 13:27:01.198187 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:27:01.198164    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea\": container with ID starting with b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea not found: ID does not exist" containerID="b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea"
Oct 31 13:27:01.198255 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:01.198199    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea"} err="failed to get container status \"b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea\": rpc error: code = NotFound desc = could not find container \"b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea\": container with ID starting with b12f8a0c3364f8e600d1370699e21583733de9f0f4910986122684b1375e42ea not found: ID does not exist"
Oct 31 13:27:01.209233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:01.209205    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"]
Oct 31 13:27:01.211628 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:01.211607    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-2c64dd/simulator-single-main-2c64dd-kserve-77ff975b99-sbvng"]
Oct 31 13:27:02.704587 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:02.704553    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" path="/var/lib/kubelet/pods/a799f8c8-7b22-4d82-a6ec-f32e9e63d14b/volumes"
Oct 31 13:27:18.245753 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:27:18.245698    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:27:22.938614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:22.938583    5787 scope.go:122] "RemoveContainer" containerID="24590cc19d3f60d92389d72c83c4af8c55103953f0aafdc9e76b732ee1c8de95"
Oct 31 13:27:22.948176 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:22.948150    5787 scope.go:122] "RemoveContainer" containerID="df7f1de005d37e8291e78d1174494e3e862bd8fac36cdc4469ef6a58cb3a76fa"
Oct 31 13:27:26.179678 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.179637    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"]
Oct 31 13:27:26.180134 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.180112    5787 cpu_manager.go:414] "containerMap: removing container" podUID="0ee08e21-0017-4597-86e5-d97ec5f8483b" containerName="revoke-keys"
Oct 31 13:27:26.180181 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.180140    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ee08e21-0017-4597-86e5-d97ec5f8483b" containerName="revoke-keys"
Oct 31 13:27:26.180216 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.180201    5787 cpu_manager.go:414] "containerMap: removing container" podUID="a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" containerName="main"
Oct 31 13:27:26.180216 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.180211    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" containerName="main"
Oct 31 13:27:26.180300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.180290    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="a799f8c8-7b22-4d82-a6ec-f32e9e63d14b" containerName="main"
Oct 31 13:27:26.180336 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.180305    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ee08e21-0017-4597-86e5-d97ec5f8483b" containerName="revoke-keys"
Oct 31 13:27:26.185230 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.185202    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.194888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.193403    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tcfhh\""
Oct 31 13:27:26.194888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.193489    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-19627735-kserve-self-signed-certs\""
Oct 31 13:27:26.195431 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.194952    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"]
Oct 31 13:27:26.253925 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.253881    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 13:27:26.253925 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.253881    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 13:27:26.286338 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.286302    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-home\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.286506 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.286360    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-dshm\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.286575 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.286515    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-tmp-dir\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.286621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.286588    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5630b90-5d86-4f6e-a738-11eeac11148f-tls-certs\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.286675 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.286628    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-model-cache\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.286782 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.286763    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2jwk\" (UniqueName: \"kubernetes.io/projected/c5630b90-5d86-4f6e-a738-11eeac11148f-kube-api-access-j2jwk\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.387613 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.387581    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-home\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.387857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.387635    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-dshm\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.387857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.387668    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-tmp-dir\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.387857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.387732    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5630b90-5d86-4f6e-a738-11eeac11148f-tls-certs\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.387857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.387758    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-model-cache\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.387857 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.387788    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2jwk\" (UniqueName: \"kubernetes.io/projected/c5630b90-5d86-4f6e-a738-11eeac11148f-kube-api-access-j2jwk\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.388153 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.388100    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-home\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.388220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.388144    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-tmp-dir\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.388220 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.388181    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-model-cache\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.390223 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.390201    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-dshm\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.390451 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.390431    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5630b90-5d86-4f6e-a738-11eeac11148f-tls-certs\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.403391 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.403356    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2jwk\" (UniqueName: \"kubernetes.io/projected/c5630b90-5d86-4f6e-a738-11eeac11148f-kube-api-access-j2jwk\") pod \"e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") " pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.500235 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.500191    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:26.776099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:26.776069    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"]
Oct 31 13:27:26.778125 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:27:26.778096    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5630b90_5d86_4f6e_a738_11eeac11148f.slice/crio-c55174463143968b437b4bda158825eec53e2b22071a82f70b6ea6cf5abde995 WatchSource:0}: Error finding container c55174463143968b437b4bda158825eec53e2b22071a82f70b6ea6cf5abde995: Status 404 returned error can't find the container with id c55174463143968b437b4bda158825eec53e2b22071a82f70b6ea6cf5abde995
Oct 31 13:27:27.341299 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:27.341254    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h" event={"ID":"c5630b90-5d86-4f6e-a738-11eeac11148f","Type":"ContainerStarted","Data":"840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87"}
Oct 31 13:27:27.341299 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:27.341311    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h" event={"ID":"c5630b90-5d86-4f6e-a738-11eeac11148f","Type":"ContainerStarted","Data":"c55174463143968b437b4bda158825eec53e2b22071a82f70b6ea6cf5abde995"}
Oct 31 13:27:27.361019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:27.360901    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h" podStartSLOduration=1.360881319 podStartE2EDuration="1.360881319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:27.360612213 +0000 UTC m=+1626.776554054" watchObservedRunningTime="2026-10-31 13:27:27.360881319 +0000 UTC m=+1626.776823164"
Oct 31 13:27:27.500978 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:27.500942    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:36.501254 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:36.501204    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:36.513837 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:36.513806    5787 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:37.433758 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:37.433723    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:27:48.247499 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:27:48.247461    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:27:59.027216 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:59.027167    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"]
Oct 31 13:27:59.032065 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:27:59.032038    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15b26686-zj4jf"]
Oct 31 13:28:00.704024 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:00.703990    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e9893a2-da1a-410a-85a7-14565155b107" path="/var/lib/kubelet/pods/4e9893a2-da1a-410a-85a7-14565155b107/volumes"
Oct 31 13:28:05.075388 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:05.075339    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"]
Oct 31 13:28:18.245218 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:28:18.245182    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:28:20.109938 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.109886    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h" podUID="c5630b90-5d86-4f6e-a738-11eeac11148f" containerName="main" containerID="cri-o://840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87" gracePeriod=45
Oct 31 13:28:20.597730 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.596955    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:28:20.712459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.710393    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-dshm\") pod \"c5630b90-5d86-4f6e-a738-11eeac11148f\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") "
Oct 31 13:28:20.712459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.710457    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5630b90-5d86-4f6e-a738-11eeac11148f-kube-api-access-j2jwk\" (UniqueName: \"kubernetes.io/projected/c5630b90-5d86-4f6e-a738-11eeac11148f-kube-api-access-j2jwk\") pod \"c5630b90-5d86-4f6e-a738-11eeac11148f\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") "
Oct 31 13:28:20.712459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.710500    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-home\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-home\") pod \"c5630b90-5d86-4f6e-a738-11eeac11148f\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") "
Oct 31 13:28:20.712459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.710538    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5630b90-5d86-4f6e-a738-11eeac11148f-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5630b90-5d86-4f6e-a738-11eeac11148f-tls-certs\") pod \"c5630b90-5d86-4f6e-a738-11eeac11148f\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") "
Oct 31 13:28:20.712459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.710584    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-model-cache\") pod \"c5630b90-5d86-4f6e-a738-11eeac11148f\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") "
Oct 31 13:28:20.712459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.710696    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-tmp-dir\") pod \"c5630b90-5d86-4f6e-a738-11eeac11148f\" (UID: \"c5630b90-5d86-4f6e-a738-11eeac11148f\") "
Oct 31 13:28:20.712459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.712060    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-home" pod "c5630b90-5d86-4f6e-a738-11eeac11148f" (UID: "c5630b90-5d86-4f6e-a738-11eeac11148f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:28:20.712459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.712361    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-model-cache" pod "c5630b90-5d86-4f6e-a738-11eeac11148f" (UID: "c5630b90-5d86-4f6e-a738-11eeac11148f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:28:20.712459 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.712422    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-tmp-dir" pod "c5630b90-5d86-4f6e-a738-11eeac11148f" (UID: "c5630b90-5d86-4f6e-a738-11eeac11148f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:28:20.715066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.714550    5787 generic.go:356] "Generic (PLEG): container finished" podID="c5630b90-5d86-4f6e-a738-11eeac11148f" containerID="840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87" exitCode=0
Oct 31 13:28:20.715066 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.714660    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"
Oct 31 13:28:20.716171 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.716136    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5630b90-5d86-4f6e-a738-11eeac11148f-kube-api-access-j2jwk" pod "c5630b90-5d86-4f6e-a738-11eeac11148f" (UID: "c5630b90-5d86-4f6e-a738-11eeac11148f"). InnerVolumeSpecName "kube-api-access-j2jwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:28:20.718233 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.718193    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5630b90-5d86-4f6e-a738-11eeac11148f-tls-certs" pod "c5630b90-5d86-4f6e-a738-11eeac11148f" (UID: "c5630b90-5d86-4f6e-a738-11eeac11148f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:28:20.719882 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.719846    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-dshm" pod "c5630b90-5d86-4f6e-a738-11eeac11148f" (UID: "c5630b90-5d86-4f6e-a738-11eeac11148f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 13:28:20.722564 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.722537    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h" event={"ID":"c5630b90-5d86-4f6e-a738-11eeac11148f","Type":"ContainerDied","Data":"840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87"}
Oct 31 13:28:20.722644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.722572    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h" event={"ID":"c5630b90-5d86-4f6e-a738-11eeac11148f","Type":"ContainerDied","Data":"c55174463143968b437b4bda158825eec53e2b22071a82f70b6ea6cf5abde995"}
Oct 31 13:28:20.722644 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.722595    5787 scope.go:122] "RemoveContainer" containerID="840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87"
Oct 31 13:28:20.807807 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.805884    5787 scope.go:122] "RemoveContainer" containerID="840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87"
Oct 31 13:28:20.807807 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:28:20.807517    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87\": container with ID starting with 840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87 not found: ID does not exist" containerID="840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87"
Oct 31 13:28:20.807807 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.807558    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87"} err="failed to get container status \"840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87\": rpc error: code = NotFound desc = could not find container \"840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87\": container with ID starting with 840caf5bc1d96eb035a754248c094440b523f303e34066bc82de380acd3eea87 not found: ID does not exist"
Oct 31 13:28:20.816465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.816340    5787 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-home\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:20.816465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.816372    5787 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5630b90-5d86-4f6e-a738-11eeac11148f-tls-certs\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:20.816465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.816387    5787 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-model-cache\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:20.816465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.816404    5787 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-tmp-dir\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:20.816465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.816418    5787 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5630b90-5d86-4f6e-a738-11eeac11148f-dshm\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:20.816465 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:20.816432    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2jwk\" (UniqueName: \"kubernetes.io/projected/c5630b90-5d86-4f6e-a738-11eeac11148f-kube-api-access-j2jwk\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:28:21.033257 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:21.033179    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"]
Oct 31 13:28:21.041162 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:21.041102    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h"]
Oct 31 13:28:21.401965 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:21.401851    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-72d7r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:28:21.402522 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:21.402491    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-72d7r" podUID="2b850a6e-358a-4baa-8730-9b569d126952" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:21.424300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:21.424261    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-19627735-kserve-6d74cdc645-sr76h" podUID="c5630b90-5d86-4f6e-a738-11eeac11148f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.149:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:21.539931 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:21.539883    5787 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f5667f86c-8h4dd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 13:28:21.540138 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:21.539958    5787 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f5667f86c-8h4dd" podUID="80d56747-87ac-4de5-a195-7d50afcee740" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 13:28:22.706769 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:22.706700    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5630b90-5d86-4f6e-a738-11eeac11148f" path="/var/lib/kubelet/pods/c5630b90-5d86-4f6e-a738-11eeac11148f/volumes"
Oct 31 13:28:22.977074 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:22.976977    5787 scope.go:122] "RemoveContainer" containerID="226890a5729da30bef99730117a54283b8a9ad9c477e552c7d9c498b624063dd"
Oct 31 13:28:48.245321 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:28:48.245278    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:28:53.067957 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.067919    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"]
Oct 31 13:28:53.068515 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.068495    5787 cpu_manager.go:414] "containerMap: removing container" podUID="c5630b90-5d86-4f6e-a738-11eeac11148f" containerName="main"
Oct 31 13:28:53.068603 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.068518    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5630b90-5d86-4f6e-a738-11eeac11148f" containerName="main"
Oct 31 13:28:53.068665 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.068623    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5630b90-5d86-4f6e-a738-11eeac11148f" containerName="main"
Oct 31 13:28:53.071721 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.071685    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:28:53.083334 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.083313    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-351178df-dockercfg-9kd8t\""
Oct 31 13:28:53.086784 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.086758    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"]
Oct 31 13:28:53.141019 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.140984    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-351178df\""
Oct 31 13:28:53.144440 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.144418    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8ldw\" (UniqueName: \"kubernetes.io/projected/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-kube-api-access-k8ldw\") pod \"payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp\" (UID: \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:28:53.144562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.144545    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp\" (UID: \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:28:53.245377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.245345    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8ldw\" (UniqueName: \"kubernetes.io/projected/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-kube-api-access-k8ldw\") pod \"payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp\" (UID: \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:28:53.245561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.245437    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp\" (UID: \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:28:53.246127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.246109    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp\" (UID: \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:28:53.262187 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.262162    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8ldw\" (UniqueName: \"kubernetes.io/projected/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-kube-api-access-k8ldw\") pod \"payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp\" (UID: \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:28:53.384995 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.384915    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:28:53.674348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.674277    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"]
Oct 31 13:28:53.677554 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:28:53.677523    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3aadbc5_2426_4f8e_9cfa_26c4a8919ec5.slice/crio-5ccbf534a2a22a50b7594c90af0fa01026d4648aeabbebc2f293b3be1e8e4a71 WatchSource:0}: Error finding container 5ccbf534a2a22a50b7594c90af0fa01026d4648aeabbebc2f293b3be1e8e4a71: Status 404 returned error can't find the container with id 5ccbf534a2a22a50b7594c90af0fa01026d4648aeabbebc2f293b3be1e8e4a71
Oct 31 13:28:53.870836 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.870796    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp" event={"ID":"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5","Type":"ContainerStarted","Data":"ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae"}
Oct 31 13:28:53.871021 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.870856    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp" event={"ID":"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5","Type":"ContainerStarted","Data":"5ccbf534a2a22a50b7594c90af0fa01026d4648aeabbebc2f293b3be1e8e4a71"}
Oct 31 13:28:53.871021 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.870884    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:28:53.891766 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:28:53.891721    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp" podStartSLOduration=0.891693134 podStartE2EDuration="891.693134ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:53.889329013 +0000 UTC m=+1713.305270856" watchObservedRunningTime="2026-10-31 13:28:53.891693134 +0000 UTC m=+1713.307634971"
Oct 31 13:29:04.878740 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:04.878648    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:29:07.499129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.499090    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"]
Oct 31 13:29:07.502767 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.502745    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:07.506214 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.506189    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xq47f\" (UniqueName: \"kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:07.506335 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.506230    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:07.506376 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.506333    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:07.507512 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.507492    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:29:07.507629 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.507529    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:29:07.508132 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.508105    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:29:07.508132 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.508119    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-v9mdk\""
Oct 31 13:29:07.512044 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.512019    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"]
Oct 31 13:29:07.606914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.606812    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq47f\" (UniqueName: \"kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:07.606914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.606859    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:07.606914 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.606904    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:07.607525 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.607507    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:07.609437 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:07.609415    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:07.613872 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:29:07.613854    5787 projected.go:196] Error preparing data for projected volume kube-api-access-xq47f for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 13:29:07.613955 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:29:07.613945    5787 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f podName:e49e7826-74f1-4871-9d7e-0f11d75816bc nodeName:}" failed. No retries permitted until 2026-10-31 13:29:08.113929383 +0000 UTC m=+1727.529871220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xq47f" (UniqueName: "kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f") pod "maas-api-revoke-keys-e2e-ait-351178df-t9lr4" (UID: "e49e7826-74f1-4871-9d7e-0f11d75816bc") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 13:29:08.214032 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:08.213462    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq47f\" (UniqueName: \"kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:08.218367 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:29:08.218336    5787 projected.go:196] Error preparing data for projected volume kube-api-access-xq47f for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 13:29:08.218632 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:29:08.218618    5787 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f podName:e49e7826-74f1-4871-9d7e-0f11d75816bc nodeName:}" failed. No retries permitted until 2026-10-31 13:29:09.21859419 +0000 UTC m=+1728.634536013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xq47f" (UniqueName: "kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f") pod "maas-api-revoke-keys-e2e-ait-351178df-t9lr4" (UID: "e49e7826-74f1-4871-9d7e-0f11d75816bc") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 13:29:09.228947 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:09.228913    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xq47f\" (UniqueName: \"kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:09.242375 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:09.242346    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xq47f\" (UniqueName: \"kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f\") pod \"maas-api-revoke-keys-e2e-ait-351178df-t9lr4\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:09.317474 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:09.317434    5787 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4" secret="" err="secret \"maas-api-cleanup-dockercfg-v9mdk\" not found"
Oct 31 13:29:09.317660 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:09.317492    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:09.592856 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:09.592778    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"]
Oct 31 13:29:09.596054 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:29:09.596012    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode49e7826_74f1_4871_9d7e_0f11d75816bc.slice/crio-2a8c6ddeb382572e72b2b793cc17afd35f04f7a455718d56c0f2f5dd8ab8cf5c WatchSource:0}: Error finding container 2a8c6ddeb382572e72b2b793cc17afd35f04f7a455718d56c0f2f5dd8ab8cf5c: Status 404 returned error can't find the container with id 2a8c6ddeb382572e72b2b793cc17afd35f04f7a455718d56c0f2f5dd8ab8cf5c
Oct 31 13:29:09.951087 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:09.951052    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4" event={"ID":"e49e7826-74f1-4871-9d7e-0f11d75816bc","Type":"ContainerStarted","Data":"2a8c6ddeb382572e72b2b793cc17afd35f04f7a455718d56c0f2f5dd8ab8cf5c"}
Oct 31 13:29:10.957575 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:10.957538    5787 generic.go:356] "Generic (PLEG): container finished" podID="e49e7826-74f1-4871-9d7e-0f11d75816bc" containerID="6ce594a75dfa009c31bf108d3cf59e44be688015a3f56dfcfbd52d9c8c67e512" exitCode=0
Oct 31 13:29:10.958016 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:10.957591    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4" event={"ID":"e49e7826-74f1-4871-9d7e-0f11d75816bc","Type":"ContainerDied","Data":"6ce594a75dfa009c31bf108d3cf59e44be688015a3f56dfcfbd52d9c8c67e512"}
Oct 31 13:29:12.104942 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.104910    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:12.156646 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.156611    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-xks-ca\") pod \"e49e7826-74f1-4871-9d7e-0f11d75816bc\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") "
Oct 31 13:29:12.156866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.156740    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f\" (UniqueName: \"kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f\") pod \"e49e7826-74f1-4871-9d7e-0f11d75816bc\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") "
Oct 31 13:29:12.156866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.156837    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-openshift-ca\") pod \"e49e7826-74f1-4871-9d7e-0f11d75816bc\" (UID: \"e49e7826-74f1-4871-9d7e-0f11d75816bc\") "
Oct 31 13:29:12.157517 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.157484    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-openshift-ca" pod "e49e7826-74f1-4871-9d7e-0f11d75816bc" (UID: "e49e7826-74f1-4871-9d7e-0f11d75816bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:29:12.160873 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.160815    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f" pod "e49e7826-74f1-4871-9d7e-0f11d75816bc" (UID: "e49e7826-74f1-4871-9d7e-0f11d75816bc"). InnerVolumeSpecName "kube-api-access-xq47f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:29:12.161004 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.160961    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-xks-ca" pod "e49e7826-74f1-4871-9d7e-0f11d75816bc" (UID: "e49e7826-74f1-4871-9d7e-0f11d75816bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:29:12.258371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.258279    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xq47f\" (UniqueName: \"kubernetes.io/projected/e49e7826-74f1-4871-9d7e-0f11d75816bc-kube-api-access-xq47f\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:12.258371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.258314    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:12.258371 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.258326    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e49e7826-74f1-4871-9d7e-0f11d75816bc-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:12.968002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.967914    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"
Oct 31 13:29:12.968002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.967924    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4" event={"ID":"e49e7826-74f1-4871-9d7e-0f11d75816bc","Type":"ContainerDied","Data":"2a8c6ddeb382572e72b2b793cc17afd35f04f7a455718d56c0f2f5dd8ab8cf5c"}
Oct 31 13:29:12.968002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:12.967974    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a8c6ddeb382572e72b2b793cc17afd35f04f7a455718d56c0f2f5dd8ab8cf5c"
Oct 31 13:29:17.655803 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:17.655767    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"]
Oct 31 13:29:18.136333 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:18.136301    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"]
Oct 31 13:29:18.139584 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:18.139545    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-351178df-t9lr4"]
Oct 31 13:29:18.266006 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:29:18.265970    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:29:18.705307 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:18.705273    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e49e7826-74f1-4871-9d7e-0f11d75816bc" path="/var/lib/kubelet/pods/e49e7826-74f1-4871-9d7e-0f11d75816bc/volumes"
Oct 31 13:29:22.657010 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:22.656956    5787 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp" podUID="f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5" containerName="payload-pre-processing" containerID="cri-o://ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae" gracePeriod=25
Oct 31 13:29:22.907248 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:22.907174    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:29:22.970352 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:22.970317    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-plugins-config-volume\") pod \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\" (UID: \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\") "
Oct 31 13:29:22.970522 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:22.970413    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-kube-api-access-k8ldw\" (UniqueName: \"kubernetes.io/projected/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-kube-api-access-k8ldw\") pod \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\" (UID: \"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5\") "
Oct 31 13:29:22.970697 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:22.970675    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-plugins-config-volume" pod "f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5" (UID: "f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:29:22.972624 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:22.972600    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-kube-api-access-k8ldw" pod "f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5" (UID: "f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5"). InnerVolumeSpecName "kube-api-access-k8ldw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:29:23.009017 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.008985    5787 scope.go:122] "RemoveContainer" containerID="3ac4dd607f66b96483993e90d85646aa86473feed2321ee927577357cb76b564"
Oct 31 13:29:23.017805 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.017782    5787 scope.go:122] "RemoveContainer" containerID="68c8dd5c2f734e9d2b11de47819ef330b0a736ec3660ec708f1f56d66e6ccf77"
Oct 31 13:29:23.022444 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.022413    5787 generic.go:356] "Generic (PLEG): container finished" podID="f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5" containerID="ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae" exitCode=0
Oct 31 13:29:23.022561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.022477    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"
Oct 31 13:29:23.022561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.022497    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp" event={"ID":"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5","Type":"ContainerDied","Data":"ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae"}
Oct 31 13:29:23.022561 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.022557    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp" event={"ID":"f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5","Type":"ContainerDied","Data":"5ccbf534a2a22a50b7594c90af0fa01026d4648aeabbebc2f293b3be1e8e4a71"}
Oct 31 13:29:23.022746 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.022587    5787 scope.go:122] "RemoveContainer" containerID="ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae"
Oct 31 13:29:23.031363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.031344    5787 scope.go:122] "RemoveContainer" containerID="ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae"
Oct 31 13:29:23.031649 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:29:23.031626    5787 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae\": container with ID starting with ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae not found: ID does not exist" containerID="ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae"
Oct 31 13:29:23.031766 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.031661    5787 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae"} err="failed to get container status \"ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae\": rpc error: code = NotFound desc = could not find container \"ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae\": container with ID starting with ba1cb3ec69cb25d568f8c5deeaea0b21a0c8dfe709edf37480c2d577a11ae9ae not found: ID does not exist"
Oct 31 13:29:23.061127 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.061096    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"]
Oct 31 13:29:23.067183 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.067155    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-351178df-6b6f7f7b8c-wbdsp"]
Oct 31 13:29:23.071212 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.071182    5787 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-plugins-config-volume\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:23.071212 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:23.071216    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8ldw\" (UniqueName: \"kubernetes.io/projected/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5-kube-api-access-k8ldw\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:29:24.703838 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:29:24.703793    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5" path="/var/lib/kubelet/pods/f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5/volumes"
Oct 31 13:29:48.246063 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:29:48.246020    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:30:00.144651 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.144611    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"]
Oct 31 13:30:00.145172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.145018    5787 cpu_manager.go:414] "containerMap: removing container" podUID="e49e7826-74f1-4871-9d7e-0f11d75816bc" containerName="revoke-keys"
Oct 31 13:30:00.145172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.145030    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e49e7826-74f1-4871-9d7e-0f11d75816bc" containerName="revoke-keys"
Oct 31 13:30:00.145172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.145040    5787 cpu_manager.go:414] "containerMap: removing container" podUID="f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5" containerName="payload-pre-processing"
Oct 31 13:30:00.145172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.145047    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5" containerName="payload-pre-processing"
Oct 31 13:30:00.145172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.145125    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="e49e7826-74f1-4871-9d7e-0f11d75816bc" containerName="revoke-keys"
Oct 31 13:30:00.145172 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.145133    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3aadbc5-2426-4f8e-9cfa-26c4a8919ec5" containerName="payload-pre-processing"
Oct 31 13:30:00.148015 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.147992    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:00.157939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.157908    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"]
Oct 31 13:30:00.158185 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.158167    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:30:00.158273 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.158248    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:30:00.158343 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.158291    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z5j9\""
Oct 31 13:30:00.276591 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.276552    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzhc9\" (UniqueName: \"kubernetes.io/projected/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-kube-api-access-pzhc9\") pod \"maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5\" (UID: \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:00.276591 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.276602    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5\" (UID: \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:00.378204 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.378165    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzhc9\" (UniqueName: \"kubernetes.io/projected/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-kube-api-access-pzhc9\") pod \"maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5\" (UID: \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:00.378389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.378215    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5\" (UID: \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:00.378870 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.378852    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5\" (UID: \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:00.391553 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.391525    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzhc9\" (UniqueName: \"kubernetes.io/projected/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-kube-api-access-pzhc9\") pod \"maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5\" (UID: \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:00.462123 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.462089    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:00.735606 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:00.735495    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"]
Oct 31 13:30:00.738336 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:30:00.738309    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd2f452_27c0_4b9b_a1cd_2d744cb19098.slice/crio-3ab6f8404143c330f9182957fc65e3ec09e07ac9b72e32450e2246ccdbe2dee1 WatchSource:0}: Error finding container 3ab6f8404143c330f9182957fc65e3ec09e07ac9b72e32450e2246ccdbe2dee1: Status 404 returned error can't find the container with id 3ab6f8404143c330f9182957fc65e3ec09e07ac9b72e32450e2246ccdbe2dee1
Oct 31 13:30:01.218337 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:01.218296    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" event={"ID":"6cd2f452-27c0-4b9b-a1cd-2d744cb19098","Type":"ContainerStarted","Data":"ef931b8fda62682e40d465957f2d6c35db36eabe8d9474a1cc7bcca21dc366e4"}
Oct 31 13:30:01.218761 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:01.218356    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" event={"ID":"6cd2f452-27c0-4b9b-a1cd-2d744cb19098","Type":"ContainerStarted","Data":"3ab6f8404143c330f9182957fc65e3ec09e07ac9b72e32450e2246ccdbe2dee1"}
Oct 31 13:30:01.237888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:01.237769    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" podStartSLOduration=1.237753491 podStartE2EDuration="1.237753491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:01.237172786 +0000 UTC m=+1780.653114627" watchObservedRunningTime="2026-10-31 13:30:01.237753491 +0000 UTC m=+1780.653695332"
Oct 31 13:30:02.224158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:02.224124    5787 generic.go:356] "Generic (PLEG): container finished" podID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerID="ef931b8fda62682e40d465957f2d6c35db36eabe8d9474a1cc7bcca21dc366e4" exitCode=60
Oct 31 13:30:02.224542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:02.224201    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" event={"ID":"6cd2f452-27c0-4b9b-a1cd-2d744cb19098","Type":"ContainerDied","Data":"ef931b8fda62682e40d465957f2d6c35db36eabe8d9474a1cc7bcca21dc366e4"}
Oct 31 13:30:02.224542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:02.224388    5787 scope.go:122] "RemoveContainer" containerID="ef931b8fda62682e40d465957f2d6c35db36eabe8d9474a1cc7bcca21dc366e4"
Oct 31 13:30:03.238686 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:03.238647    5787 generic.go:356] "Generic (PLEG): container finished" podID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerID="e76c984e4b1d0adc250ade36cc9d0f8ffff062351d453c5910ee457863d6f4e6" exitCode=60
Oct 31 13:30:03.239571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:03.239381    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" event={"ID":"6cd2f452-27c0-4b9b-a1cd-2d744cb19098","Type":"ContainerDied","Data":"e76c984e4b1d0adc250ade36cc9d0f8ffff062351d453c5910ee457863d6f4e6"}
Oct 31 13:30:03.239571 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:03.239458    5787 scope.go:122] "RemoveContainer" containerID="ef931b8fda62682e40d465957f2d6c35db36eabe8d9474a1cc7bcca21dc366e4"
Oct 31 13:30:03.239799 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:03.239769    5787 scope.go:122] "RemoveContainer" containerID="e76c984e4b1d0adc250ade36cc9d0f8ffff062351d453c5910ee457863d6f4e6"
Oct 31 13:30:03.240112 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:30:03.240052    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5_odh-ai-gateway-infra(6cd2f452-27c0-4b9b-a1cd-2d744cb19098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098"
Oct 31 13:30:04.246057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:04.246027    5787 scope.go:122] "RemoveContainer" containerID="e76c984e4b1d0adc250ade36cc9d0f8ffff062351d453c5910ee457863d6f4e6"
Oct 31 13:30:04.246475 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:30:04.246255    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5_odh-ai-gateway-infra(6cd2f452-27c0-4b9b-a1cd-2d744cb19098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098"
Oct 31 13:30:15.699321 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:15.699288    5787 scope.go:122] "RemoveContainer" containerID="e76c984e4b1d0adc250ade36cc9d0f8ffff062351d453c5910ee457863d6f4e6"
Oct 31 13:30:16.304268 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:16.304228    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" event={"ID":"6cd2f452-27c0-4b9b-a1cd-2d744cb19098","Type":"ContainerStarted","Data":"5a9e0689f00c1474edc85e36318dec8479f505cde01d2de4396e0dd0064e8932"}
Oct 31 13:30:17.310415 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:17.310380    5787 generic.go:356] "Generic (PLEG): container finished" podID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerID="5a9e0689f00c1474edc85e36318dec8479f505cde01d2de4396e0dd0064e8932" exitCode=60
Oct 31 13:30:17.310832 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:17.310448    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" event={"ID":"6cd2f452-27c0-4b9b-a1cd-2d744cb19098","Type":"ContainerDied","Data":"5a9e0689f00c1474edc85e36318dec8479f505cde01d2de4396e0dd0064e8932"}
Oct 31 13:30:17.310832 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:17.310537    5787 scope.go:122] "RemoveContainer" containerID="e76c984e4b1d0adc250ade36cc9d0f8ffff062351d453c5910ee457863d6f4e6"
Oct 31 13:30:17.310966 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:17.310866    5787 scope.go:122] "RemoveContainer" containerID="5a9e0689f00c1474edc85e36318dec8479f505cde01d2de4396e0dd0064e8932"
Oct 31 13:30:17.311133 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:30:17.311116    5787 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5_odh-ai-gateway-infra(6cd2f452-27c0-4b9b-a1cd-2d744cb19098)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098"
Oct 31 13:30:17.353607 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:17.353564    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"]
Oct 31 13:30:18.245247 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:30:18.245208    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:30:18.457031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:18.456994    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:18.463924 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:18.463896    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-maas-api-service-ca\") pod \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\" (UID: \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\") "
Oct 31 13:30:18.464058 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:18.464011    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-kube-api-access-pzhc9\" (UniqueName: \"kubernetes.io/projected/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-kube-api-access-pzhc9\") pod \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\" (UID: \"6cd2f452-27c0-4b9b-a1cd-2d744cb19098\") "
Oct 31 13:30:18.464234 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:18.464214    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-maas-api-service-ca" pod "6cd2f452-27c0-4b9b-a1cd-2d744cb19098" (UID: "6cd2f452-27c0-4b9b-a1cd-2d744cb19098"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:30:18.466173 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:18.466148    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-kube-api-access-pzhc9" pod "6cd2f452-27c0-4b9b-a1cd-2d744cb19098" (UID: "6cd2f452-27c0-4b9b-a1cd-2d744cb19098"). InnerVolumeSpecName "kube-api-access-pzhc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:30:18.565194 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:18.565109    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzhc9\" (UniqueName: \"kubernetes.io/projected/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-kube-api-access-pzhc9\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:18.565194 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:18.565140    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6cd2f452-27c0-4b9b-a1cd-2d744cb19098-maas-api-service-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:30:19.324807 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:19.324771    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"
Oct 31 13:30:19.324994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:19.324778    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5" event={"ID":"6cd2f452-27c0-4b9b-a1cd-2d744cb19098","Type":"ContainerDied","Data":"3ab6f8404143c330f9182957fc65e3ec09e07ac9b72e32450e2246ccdbe2dee1"}
Oct 31 13:30:19.324994 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:19.324910    5787 scope.go:122] "RemoveContainer" containerID="5a9e0689f00c1474edc85e36318dec8479f505cde01d2de4396e0dd0064e8932"
Oct 31 13:30:19.345804 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:19.345775    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"]
Oct 31 13:30:19.354811 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:19.354783    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-authcfg-http-9c81e5-29890890-rxjr5"]
Oct 31 13:30:20.704466 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:30:20.704427    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" path="/var/lib/kubelet/pods/6cd2f452-27c0-4b9b-a1cd-2d744cb19098/volumes"
Oct 31 13:30:48.244433 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:30:48.244390    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:31:04.737332 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737295    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"]
Oct 31 13:31:04.739891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737677    5787 cpu_manager.go:414] "containerMap: removing container" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerName="cleanup"
Oct 31 13:31:04.739891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737687    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerName="cleanup"
Oct 31 13:31:04.739891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737727    5787 cpu_manager.go:414] "containerMap: removing container" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerName="cleanup"
Oct 31 13:31:04.739891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737732    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerName="cleanup"
Oct 31 13:31:04.739891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737739    5787 cpu_manager.go:414] "containerMap: removing container" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerName="cleanup"
Oct 31 13:31:04.739891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737744    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerName="cleanup"
Oct 31 13:31:04.739891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737805    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerName="cleanup"
Oct 31 13:31:04.739891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737814    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerName="cleanup"
Oct 31 13:31:04.739891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.737821    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd2f452-27c0-4b9b-a1cd-2d744cb19098" containerName="cleanup"
Oct 31 13:31:04.740933 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.740917    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:04.750360 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.750331    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:31:04.751136 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.751121    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z5j9\""
Oct 31 13:31:04.753542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.753516    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"]
Oct 31 13:31:04.837640 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.837606    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:31:04.837640 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.837613    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:31:04.845928 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.845905    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:04.845992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.845958    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm6r7\" (UniqueName: \"kubernetes.io/projected/76951ebe-b150-473a-bb7b-ff3b414f2a66-kube-api-access-cm6r7\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:04.846057 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.846041    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:04.947622 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.947584    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm6r7\" (UniqueName: \"kubernetes.io/projected/76951ebe-b150-473a-bb7b-ff3b414f2a66-kube-api-access-cm6r7\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:04.947819 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.947632    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:04.947819 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.947736    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:04.948326 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.948305    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:04.950246 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.950226    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:04.960901 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:04.960878    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm6r7\" (UniqueName: \"kubernetes.io/projected/76951ebe-b150-473a-bb7b-ff3b414f2a66-kube-api-access-cm6r7\") pod \"maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:05.053521 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:05.053437    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:05.332113 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:05.332031    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"]
Oct 31 13:31:05.335607 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:31:05.335580    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76951ebe_b150_473a_bb7b_ff3b414f2a66.slice/crio-30cd7576724ffdaa29f9b4f9eb4715d51ff8257afe32c1cc5bb3d8d8bfbb72db WatchSource:0}: Error finding container 30cd7576724ffdaa29f9b4f9eb4715d51ff8257afe32c1cc5bb3d8d8bfbb72db: Status 404 returned error can't find the container with id 30cd7576724ffdaa29f9b4f9eb4715d51ff8257afe32c1cc5bb3d8d8bfbb72db
Oct 31 13:31:05.528243 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:05.528206    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29" event={"ID":"76951ebe-b150-473a-bb7b-ff3b414f2a66","Type":"ContainerStarted","Data":"30cd7576724ffdaa29f9b4f9eb4715d51ff8257afe32c1cc5bb3d8d8bfbb72db"}
Oct 31 13:31:06.536221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:06.536125    5787 generic.go:356] "Generic (PLEG): container finished" podID="76951ebe-b150-473a-bb7b-ff3b414f2a66" containerID="e9d0ef1f16fb3c641caf03f6c99d815942af8ab488c72c75644e6eeeee4b7ce7" exitCode=0
Oct 31 13:31:06.536622 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:06.536211    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29" event={"ID":"76951ebe-b150-473a-bb7b-ff3b414f2a66","Type":"ContainerDied","Data":"e9d0ef1f16fb3c641caf03f6c99d815942af8ab488c72c75644e6eeeee4b7ce7"}
Oct 31 13:31:07.672505 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.672476    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:07.773149 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.773112    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76951ebe-b150-473a-bb7b-ff3b414f2a66-kube-api-access-cm6r7\" (UniqueName: \"kubernetes.io/projected/76951ebe-b150-473a-bb7b-ff3b414f2a66-kube-api-access-cm6r7\") pod \"76951ebe-b150-473a-bb7b-ff3b414f2a66\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") "
Oct 31 13:31:07.773345 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.773163    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-xks-ca\") pod \"76951ebe-b150-473a-bb7b-ff3b414f2a66\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") "
Oct 31 13:31:07.773345 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.773232    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-openshift-ca\") pod \"76951ebe-b150-473a-bb7b-ff3b414f2a66\" (UID: \"76951ebe-b150-473a-bb7b-ff3b414f2a66\") "
Oct 31 13:31:07.773676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.773645    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-openshift-ca" pod "76951ebe-b150-473a-bb7b-ff3b414f2a66" (UID: "76951ebe-b150-473a-bb7b-ff3b414f2a66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:07.773897 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.773874    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:07.775534 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.775504    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-xks-ca" pod "76951ebe-b150-473a-bb7b-ff3b414f2a66" (UID: "76951ebe-b150-473a-bb7b-ff3b414f2a66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:31:07.776129 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.776101    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76951ebe-b150-473a-bb7b-ff3b414f2a66-kube-api-access-cm6r7" pod "76951ebe-b150-473a-bb7b-ff3b414f2a66" (UID: "76951ebe-b150-473a-bb7b-ff3b414f2a66"). InnerVolumeSpecName "kube-api-access-cm6r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:07.875197 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.875102    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm6r7\" (UniqueName: \"kubernetes.io/projected/76951ebe-b150-473a-bb7b-ff3b414f2a66-kube-api-access-cm6r7\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:07.875197 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:07.875140    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76951ebe-b150-473a-bb7b-ff3b414f2a66-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:08.546086 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:08.546047    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29" event={"ID":"76951ebe-b150-473a-bb7b-ff3b414f2a66","Type":"ContainerDied","Data":"30cd7576724ffdaa29f9b4f9eb4715d51ff8257afe32c1cc5bb3d8d8bfbb72db"}
Oct 31 13:31:08.546300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:08.546108    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cd7576724ffdaa29f9b4f9eb4715d51ff8257afe32c1cc5bb3d8d8bfbb72db"
Oct 31 13:31:08.546300 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:08.546057    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"
Oct 31 13:31:18.246228 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:31:18.246189    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:31:21.606562 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:21.606527    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"]
Oct 31 13:31:21.610754 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:21.610718    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-9c81e5-rhf29"]
Oct 31 13:31:22.704775 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:22.704725    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76951ebe-b150-473a-bb7b-ff3b414f2a66" path="/var/lib/kubelet/pods/76951ebe-b150-473a-bb7b-ff3b414f2a66/volumes"
Oct 31 13:31:33.047731 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:33.047133    5787 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"]
Oct 31 13:31:33.051002 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:33.050965    5787 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ab5cf0-srswb"]
Oct 31 13:31:34.704400 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:34.704367    5787 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a3b647-83c4-4e73-8460-26eb07319a62" path="/var/lib/kubelet/pods/d5a3b647-83c4-4e73-8460-26eb07319a62/volumes"
Oct 31 13:31:40.909944 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.909911    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"]
Oct 31 13:31:40.910363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.910305    5787 cpu_manager.go:414] "containerMap: removing container" podUID="76951ebe-b150-473a-bb7b-ff3b414f2a66" containerName="revoke-keys"
Oct 31 13:31:40.910363 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.910317    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76951ebe-b150-473a-bb7b-ff3b414f2a66" containerName="revoke-keys"
Oct 31 13:31:40.910447 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.910378    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="76951ebe-b150-473a-bb7b-ff3b414f2a66" containerName="revoke-keys"
Oct 31 13:31:40.914650 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.914633    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:40.919413 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.919396    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:31:40.919530 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.919515    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:31:40.919615 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.919593    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z5j9\""
Oct 31 13:31:40.922054 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.921980    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:31:40.923848 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:40.923826    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"]
Oct 31 13:31:41.018100 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.018053    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zlh\" (UniqueName: \"kubernetes.io/projected/2110b812-bb34-4a76-bac2-4cae2c1f53b0-kube-api-access-w5zlh\") pod \"maas-api-revoke-sub-f03629db-vjhqz\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.018290 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.018228    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vjhqz\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.018290 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.018264    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vjhqz\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.119585 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.119550    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vjhqz\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.119777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.119595    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vjhqz\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.119777 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.119627    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zlh\" (UniqueName: \"kubernetes.io/projected/2110b812-bb34-4a76-bac2-4cae2c1f53b0-kube-api-access-w5zlh\") pod \"maas-api-revoke-sub-f03629db-vjhqz\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.120270 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.120243    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-vjhqz\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.122120 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.122102    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-vjhqz\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.136093 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.136069    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zlh\" (UniqueName: \"kubernetes.io/projected/2110b812-bb34-4a76-bac2-4cae2c1f53b0-kube-api-access-w5zlh\") pod \"maas-api-revoke-sub-f03629db-vjhqz\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.228523 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.228493    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:41.507570 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.507543    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"]
Oct 31 13:31:41.510175 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:31:41.510143    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2110b812_bb34_4a76_bac2_4cae2c1f53b0.slice/crio-0c33858f9b3246d5a9d8d4a37cefeae45a800a3b9629ce8efd9d06e5444b1ffd WatchSource:0}: Error finding container 0c33858f9b3246d5a9d8d4a37cefeae45a800a3b9629ce8efd9d06e5444b1ffd: Status 404 returned error can't find the container with id 0c33858f9b3246d5a9d8d4a37cefeae45a800a3b9629ce8efd9d06e5444b1ffd
Oct 31 13:31:41.750649 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:41.750608    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz" event={"ID":"2110b812-bb34-4a76-bac2-4cae2c1f53b0","Type":"ContainerStarted","Data":"0c33858f9b3246d5a9d8d4a37cefeae45a800a3b9629ce8efd9d06e5444b1ffd"}
Oct 31 13:31:42.757055 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:42.757016    5787 generic.go:356] "Generic (PLEG): container finished" podID="2110b812-bb34-4a76-bac2-4cae2c1f53b0" containerID="90d29053692dc399e1fe40d33a88aecdb7b7c2d8718b13d1925567cbc29e20b0" exitCode=0
Oct 31 13:31:42.757468 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:42.757109    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz" event={"ID":"2110b812-bb34-4a76-bac2-4cae2c1f53b0","Type":"ContainerDied","Data":"90d29053692dc399e1fe40d33a88aecdb7b7c2d8718b13d1925567cbc29e20b0"}
Oct 31 13:31:43.891348 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:43.891321    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:43.947992 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:43.947943    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2110b812-bb34-4a76-bac2-4cae2c1f53b0-kube-api-access-w5zlh\" (UniqueName: \"kubernetes.io/projected/2110b812-bb34-4a76-bac2-4cae2c1f53b0-kube-api-access-w5zlh\") pod \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") "
Oct 31 13:31:43.948202 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:43.948069    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-xks-ca\") pod \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") "
Oct 31 13:31:43.948202 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:43.948105    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-openshift-ca\") pod \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\" (UID: \"2110b812-bb34-4a76-bac2-4cae2c1f53b0\") "
Oct 31 13:31:43.948542 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:43.948511    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-openshift-ca" pod "2110b812-bb34-4a76-bac2-4cae2c1f53b0" (UID: "2110b812-bb34-4a76-bac2-4cae2c1f53b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:31:43.950193 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:43.950166    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-xks-ca" pod "2110b812-bb34-4a76-bac2-4cae2c1f53b0" (UID: "2110b812-bb34-4a76-bac2-4cae2c1f53b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:31:43.950297 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:43.950228    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2110b812-bb34-4a76-bac2-4cae2c1f53b0-kube-api-access-w5zlh" pod "2110b812-bb34-4a76-bac2-4cae2c1f53b0" (UID: "2110b812-bb34-4a76-bac2-4cae2c1f53b0"). InnerVolumeSpecName "kube-api-access-w5zlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:31:44.049817 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:44.049737    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5zlh\" (UniqueName: \"kubernetes.io/projected/2110b812-bb34-4a76-bac2-4cae2c1f53b0-kube-api-access-w5zlh\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:44.049817 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:44.049767    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:44.049817 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:44.049777    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2110b812-bb34-4a76-bac2-4cae2c1f53b0-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:31:44.767403 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:44.767364    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz"
Oct 31 13:31:44.767586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:44.767401    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-vjhqz" event={"ID":"2110b812-bb34-4a76-bac2-4cae2c1f53b0","Type":"ContainerDied","Data":"0c33858f9b3246d5a9d8d4a37cefeae45a800a3b9629ce8efd9d06e5444b1ffd"}
Oct 31 13:31:44.767586 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:44.767455    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c33858f9b3246d5a9d8d4a37cefeae45a800a3b9629ce8efd9d06e5444b1ffd"
Oct 31 13:31:48.249467 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:31:48.249428    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:31:56.997496 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:56.997460    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"]
Oct 31 13:31:56.997927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:56.997877    5787 cpu_manager.go:414] "containerMap: removing container" podUID="2110b812-bb34-4a76-bac2-4cae2c1f53b0" containerName="revoke-keys"
Oct 31 13:31:56.997927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:56.997890    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2110b812-bb34-4a76-bac2-4cae2c1f53b0" containerName="revoke-keys"
Oct 31 13:31:56.998027 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:56.997983    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="2110b812-bb34-4a76-bac2-4cae2c1f53b0" containerName="revoke-keys"
Oct 31 13:31:57.001137 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.001108    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.006031 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.006009    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:31:57.006309 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.006285    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:31:57.007237 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.007219    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z5j9\""
Oct 31 13:31:57.007382 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.007369    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:31:57.019271 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.019240    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"]
Oct 31 13:31:57.105174 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.105137    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx2gk\" (UniqueName: \"kubernetes.io/projected/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-kube-api-access-sx2gk\") pod \"maas-api-revoke-sub-0165d7d2-w5xss\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.105174 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.105175    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5xss\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.105389 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.105194    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5xss\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.206750 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.206667    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx2gk\" (UniqueName: \"kubernetes.io/projected/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-kube-api-access-sx2gk\") pod \"maas-api-revoke-sub-0165d7d2-w5xss\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.206750 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.206760    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5xss\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.207036 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.206786    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5xss\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.207409 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.207385    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5xss\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.209407 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.209377    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-w5xss\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.220032 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.220011    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx2gk\" (UniqueName: \"kubernetes.io/projected/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-kube-api-access-sx2gk\") pod \"maas-api-revoke-sub-0165d7d2-w5xss\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.314119 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.314050    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:31:57.589377 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.589358    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"]
Oct 31 13:31:57.591753 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:31:57.591723    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33c4c04a_c8fe_45a2_9a3d_d3389ac7a81f.slice/crio-bb410d23869d566e4da67b7c7012f55044d3de6f861350dd1d3aed8e0bfba58c WatchSource:0}: Error finding container bb410d23869d566e4da67b7c7012f55044d3de6f861350dd1d3aed8e0bfba58c: Status 404 returned error can't find the container with id bb410d23869d566e4da67b7c7012f55044d3de6f861350dd1d3aed8e0bfba58c
Oct 31 13:31:57.827725 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:57.827662    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss" event={"ID":"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f","Type":"ContainerStarted","Data":"bb410d23869d566e4da67b7c7012f55044d3de6f861350dd1d3aed8e0bfba58c"}
Oct 31 13:31:58.839424 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:58.839388    5787 generic.go:356] "Generic (PLEG): container finished" podID="33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f" containerID="1749f77aa9c8772a62a847058d3838a886fd84687e334466854bc5dff4279175" exitCode=0
Oct 31 13:31:58.839833 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:58.839473    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss" event={"ID":"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f","Type":"ContainerDied","Data":"1749f77aa9c8772a62a847058d3838a886fd84687e334466854bc5dff4279175"}
Oct 31 13:31:59.971893 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:31:59.971864    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:32:00.132079 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.131984    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-xks-ca\") pod \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") "
Oct 31 13:32:00.132241 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.132083    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-kube-api-access-sx2gk\" (UniqueName: \"kubernetes.io/projected/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-kube-api-access-sx2gk\") pod \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") "
Oct 31 13:32:00.132303 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.132247    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-openshift-ca\") pod \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\" (UID: \"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f\") "
Oct 31 13:32:00.132597 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.132574    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-openshift-ca" pod "33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f" (UID: "33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:32:00.134311 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.134281    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-xks-ca" pod "33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f" (UID: "33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:32:00.134395 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.134311    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-kube-api-access-sx2gk" pod "33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f" (UID: "33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f"). InnerVolumeSpecName "kube-api-access-sx2gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:32:00.233204 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.233151    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx2gk\" (UniqueName: \"kubernetes.io/projected/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-kube-api-access-sx2gk\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:00.233204 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.233197    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:00.233204 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.233207    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:00.850121 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.850084    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss" event={"ID":"33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f","Type":"ContainerDied","Data":"bb410d23869d566e4da67b7c7012f55044d3de6f861350dd1d3aed8e0bfba58c"}
Oct 31 13:32:00.850329 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.850132    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb410d23869d566e4da67b7c7012f55044d3de6f861350dd1d3aed8e0bfba58c"
Oct 31 13:32:00.850329 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:00.850132    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-w5xss"
Oct 31 13:32:07.829514 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.829478    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"]
Oct 31 13:32:07.830316 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.830290    5787 cpu_manager.go:414] "containerMap: removing container" podUID="33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f" containerName="revoke-keys"
Oct 31 13:32:07.830316 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.830318    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f" containerName="revoke-keys"
Oct 31 13:32:07.830527 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.830450    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f" containerName="revoke-keys"
Oct 31 13:32:07.833638 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.833621    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:07.842723 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.842674    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"]
Oct 31 13:32:07.843252 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.843229    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 13:32:07.843370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.843232    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 13:32:07.843370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.843292    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 13:32:07.843370 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:07.843358    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-4z5j9\""
Oct 31 13:32:08.012640 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.012592    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck5dd\" (UniqueName: \"kubernetes.io/projected/74256a45-c6a4-42ef-b193-220f8b8e3224-kube-api-access-ck5dd\") pod \"maas-api-revoke-sub-6b19338f-ldc85\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.012866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.012676    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ldc85\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.012866 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.012772    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ldc85\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.113940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.113853    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ldc85\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.113940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.113897    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ldc85\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.114122 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.113963    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck5dd\" (UniqueName: \"kubernetes.io/projected/74256a45-c6a4-42ef-b193-220f8b8e3224-kube-api-access-ck5dd\") pod \"maas-api-revoke-sub-6b19338f-ldc85\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.114582 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.114562    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-ldc85\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.116326 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.116304    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-ldc85\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.128021 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.127994    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck5dd\" (UniqueName: \"kubernetes.io/projected/74256a45-c6a4-42ef-b193-220f8b8e3224-kube-api-access-ck5dd\") pod \"maas-api-revoke-sub-6b19338f-ldc85\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.148979 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.148954    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:08.429926 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.428558    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"]
Oct 31 13:32:08.446396 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:32:08.446356    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74256a45_c6a4_42ef_b193_220f8b8e3224.slice/crio-5b67dfd8508aca48ded20f45cebb218f85b353d7bd8f56775275e8c02aab1acf WatchSource:0}: Error finding container 5b67dfd8508aca48ded20f45cebb218f85b353d7bd8f56775275e8c02aab1acf: Status 404 returned error can't find the container with id 5b67dfd8508aca48ded20f45cebb218f85b353d7bd8f56775275e8c02aab1acf
Oct 31 13:32:08.888329 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.888283    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85" event={"ID":"74256a45-c6a4-42ef-b193-220f8b8e3224","Type":"ContainerStarted","Data":"a5347c059ce90b435223593c49b5f01e83b912ad0ec2e054a5a754a9b00c1682"}
Oct 31 13:32:08.888755 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.888339    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85" event={"ID":"74256a45-c6a4-42ef-b193-220f8b8e3224","Type":"ContainerStarted","Data":"5b67dfd8508aca48ded20f45cebb218f85b353d7bd8f56775275e8c02aab1acf"}
Oct 31 13:32:08.920621 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:08.920563    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85" podStartSLOduration=1.920547445 podStartE2EDuration="1.920547445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:08.919423868 +0000 UTC m=+1908.335365708" watchObservedRunningTime="2026-10-31 13:32:08.920547445 +0000 UTC m=+1908.336489286"
Oct 31 13:32:09.893940 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:09.893900    5787 generic.go:356] "Generic (PLEG): container finished" podID="74256a45-c6a4-42ef-b193-220f8b8e3224" containerID="a5347c059ce90b435223593c49b5f01e83b912ad0ec2e054a5a754a9b00c1682" exitCode=0
Oct 31 13:32:09.894339 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:09.893966    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85" event={"ID":"74256a45-c6a4-42ef-b193-220f8b8e3224","Type":"ContainerDied","Data":"a5347c059ce90b435223593c49b5f01e83b912ad0ec2e054a5a754a9b00c1682"}
Oct 31 13:32:11.031983 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.031956    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:11.036951 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.036928    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-openshift-ca\") pod \"74256a45-c6a4-42ef-b193-220f8b8e3224\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") "
Oct 31 13:32:11.037053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.036980    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74256a45-c6a4-42ef-b193-220f8b8e3224-kube-api-access-ck5dd\" (UniqueName: \"kubernetes.io/projected/74256a45-c6a4-42ef-b193-220f8b8e3224-kube-api-access-ck5dd\") pod \"74256a45-c6a4-42ef-b193-220f8b8e3224\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") "
Oct 31 13:32:11.037053 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.037038    5787 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-xks-ca\") pod \"74256a45-c6a4-42ef-b193-220f8b8e3224\" (UID: \"74256a45-c6a4-42ef-b193-220f8b8e3224\") "
Oct 31 13:32:11.037333 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.037306    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-openshift-ca" pod "74256a45-c6a4-42ef-b193-220f8b8e3224" (UID: "74256a45-c6a4-42ef-b193-220f8b8e3224"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 13:32:11.039158 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.039134    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-xks-ca" pod "74256a45-c6a4-42ef-b193-220f8b8e3224" (UID: "74256a45-c6a4-42ef-b193-220f8b8e3224"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 13:32:11.039255 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.039232    5787 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74256a45-c6a4-42ef-b193-220f8b8e3224-kube-api-access-ck5dd" pod "74256a45-c6a4-42ef-b193-220f8b8e3224" (UID: "74256a45-c6a4-42ef-b193-220f8b8e3224"). InnerVolumeSpecName "kube-api-access-ck5dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 13:32:11.137884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.137849    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-openshift-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:11.137884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.137880    5787 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck5dd\" (UniqueName: \"kubernetes.io/projected/74256a45-c6a4-42ef-b193-220f8b8e3224-kube-api-access-ck5dd\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:11.137884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.137893    5787 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74256a45-c6a4-42ef-b193-220f8b8e3224-maas-api-xks-ca\") on node \"ip-10-0-141-83.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 13:32:11.904801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.904770    5787 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85"
Oct 31 13:32:11.904801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.904781    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-ldc85" event={"ID":"74256a45-c6a4-42ef-b193-220f8b8e3224","Type":"ContainerDied","Data":"5b67dfd8508aca48ded20f45cebb218f85b353d7bd8f56775275e8c02aab1acf"}
Oct 31 13:32:11.905079 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:11.904827    5787 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b67dfd8508aca48ded20f45cebb218f85b353d7bd8f56775275e8c02aab1acf"
Oct 31 13:32:18.245948 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:32:18.245899    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:32:29.927342 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:29.927303    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9klng_dc065dda-a2b2-4b71-a8f8-8cc9298e40b7/manager/0.log"
Oct 31 13:32:30.331789 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:30.331754    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xbhb_5802a4b8-1826-4452-aab5-f8d2b2bc95a1/manager/0.log"
Oct 31 13:32:32.266267 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:32.266227    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-w5xss_33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f/revoke-keys/0.log"
Oct 31 13:32:32.788307 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:32.788270    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ldc85_74256a45-c6a4-42ef-b193-220f8b8e3224/revoke-keys/0.log"
Oct 31 13:32:33.144614 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:33.144528    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-vjhqz_2110b812-bb34-4a76-bac2-4cae2c1f53b0/revoke-keys/0.log"
Oct 31 13:32:35.046159 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:35.046129    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gc6kh_e5964761-efa2-4a50-82c8-ae9a984603a9/manager/0.log"
Oct 31 13:32:35.403676 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:35.403599    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pfgg8_4d80a741-9910-40ee-90a7-7ef3a71d1c55/kuadrant-console-plugin/0.log"
Oct 31 13:32:36.923038 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:36.923008    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9bt9l_d192d1e3-c83f-4e9e-979f-684a6b98cc73/discovery/0.log"
Oct 31 13:32:37.857753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:37.857694    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-72d7r_2b850a6e-358a-4baa-8730-9b569d126952/istio-proxy/0.log"
Oct 31 13:32:38.042820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:38.042788    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8h4dd_80d56747-87ac-4de5-a195-7d50afcee740/istio-proxy/0.log"
Oct 31 13:32:38.754308 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:38.754280    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-rz8cm_d95c8381-0aff-448c-9422-d3693062872f/istio-proxy/0.log"
Oct 31 13:32:48.245922 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:32:48.245880    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:32:52.292795 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:52.292756    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9klng_dc065dda-a2b2-4b71-a8f8-8cc9298e40b7/manager/0.log"
Oct 31 13:32:52.668939 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:52.668858    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xbhb_5802a4b8-1826-4452-aab5-f8d2b2bc95a1/manager/0.log"
Oct 31 13:32:54.494000 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:54.493963    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-w5xss_33c4c04a-c8fe-45a2-9a3d-d3389ac7a81f/revoke-keys/0.log"
Oct 31 13:32:54.846210 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:54.846123    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-ldc85_74256a45-c6a4-42ef-b193-220f8b8e3224/revoke-keys/0.log"
Oct 31 13:32:55.203776 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:55.203742    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-vjhqz_2110b812-bb34-4a76-bac2-4cae2c1f53b0/revoke-keys/0.log"
Oct 31 13:32:56.768278 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:56.768242    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gc6kh_e5964761-efa2-4a50-82c8-ae9a984603a9/manager/0.log"
Oct 31 13:32:57.104869 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:57.104790    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pfgg8_4d80a741-9910-40ee-90a7-7ef3a71d1c55/kuadrant-console-plugin/0.log"
Oct 31 13:32:58.424385 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:58.424356    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9bt9l_d192d1e3-c83f-4e9e-979f-684a6b98cc73/discovery/0.log"
Oct 31 13:32:59.346922 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:59.346891    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-72d7r_2b850a6e-358a-4baa-8730-9b569d126952/istio-proxy/0.log"
Oct 31 13:32:59.540593 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:32:59.540556    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8h4dd_80d56747-87ac-4de5-a195-7d50afcee740/istio-proxy/0.log"
Oct 31 13:33:00.283827 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:33:00.283790    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-rz8cm_d95c8381-0aff-448c-9422-d3693062872f/istio-proxy/0.log"
Oct 31 13:33:18.246427 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:33:18.246382    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:33:23.074884 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:33:23.074849    5787 scope.go:122] "RemoveContainer" containerID="41c92b8e7be0eb1658c4affe015bfafb7f67be4ec8f217d7d033608a7ba86807"
Oct 31 13:33:23.083876 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:33:23.083854    5787 scope.go:122] "RemoveContainer" containerID="547ddd1ad7beb4c3a6ec3127aa976b48edd068af2368d1a952062786101446b7"
Oct 31 13:33:48.246947 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:33:48.246863    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:34:18.245648 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:34:18.245609    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:34:40.046530 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:40.046498    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-c6hpz_97d66a46-02bb-4220-a78c-850e615072ca/global-pull-secret-syncer/0.log"
Oct 31 13:34:40.410917 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:40.410839    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-phbg5_686b38d0-13b9-4d2c-a1cf-5b8053804350/konnectivity-agent/0.log"
Oct 31 13:34:40.538335 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:40.538307    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-83.us-east-2.compute.internal_7ebdbb4e6bd986d2f09f581a50c7d1f8/haproxy/0.log"
Oct 31 13:34:47.707594 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:47.707563    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gc6kh_e5964761-efa2-4a50-82c8-ae9a984603a9/manager/0.log"
Oct 31 13:34:47.793038 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:47.793006    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-pfgg8_4d80a741-9910-40ee-90a7-7ef3a71d1c55/kuadrant-console-plugin/0.log"
Oct 31 13:34:48.247042 ip-10-0-141-83 kubenswrapper[5787]: E1031 13:34:48.247007    5787 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nkxs6" containerName="extractor"
Oct 31 13:34:49.140820 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.140790    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d86fc01-6e17-4d9d-9e6d-eac5fe617705/alertmanager/0.log"
Oct 31 13:34:49.168753 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.168665    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d86fc01-6e17-4d9d-9e6d-eac5fe617705/config-reloader/0.log"
Oct 31 13:34:49.198891 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.198863    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d86fc01-6e17-4d9d-9e6d-eac5fe617705/kube-rbac-proxy-web/0.log"
Oct 31 13:34:49.227610 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.227584    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d86fc01-6e17-4d9d-9e6d-eac5fe617705/kube-rbac-proxy/0.log"
Oct 31 13:34:49.255813 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.255785    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d86fc01-6e17-4d9d-9e6d-eac5fe617705/kube-rbac-proxy-metric/0.log"
Oct 31 13:34:49.286499 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.286472    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d86fc01-6e17-4d9d-9e6d-eac5fe617705/prom-label-proxy/0.log"
Oct 31 13:34:49.315858 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.315830    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0d86fc01-6e17-4d9d-9e6d-eac5fe617705/init-config-reloader/0.log"
Oct 31 13:34:49.406096 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.405996    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2xmtm_57294d98-42b2-47f0-a79a-d9cf7053ff9c/kube-state-metrics/0.log"
Oct 31 13:34:49.434935 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.434912    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2xmtm_57294d98-42b2-47f0-a79a-d9cf7053ff9c/kube-rbac-proxy-main/0.log"
Oct 31 13:34:49.464099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.464058    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2xmtm_57294d98-42b2-47f0-a79a-d9cf7053ff9c/kube-rbac-proxy-self/0.log"
Oct 31 13:34:49.502089 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.502060    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b9f69b5fb-mnd5t_41aeee13-d6f6-4b22-b0d0-5c6a88038716/metrics-server/0.log"
Oct 31 13:34:49.532810 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.532746    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-sqfbr_be0506ab-10a8-4158-b7e1-62ded5ce9f11/monitoring-plugin/0.log"
Oct 31 13:34:49.770003 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.769970    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-srtvx_3814f51b-f30b-4f73-8061-ece6249a6edb/node-exporter/0.log"
Oct 31 13:34:49.797287 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.797250    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-srtvx_3814f51b-f30b-4f73-8061-ece6249a6edb/kube-rbac-proxy/0.log"
Oct 31 13:34:49.827613 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.827587    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-srtvx_3814f51b-f30b-4f73-8061-ece6249a6edb/init-textfile/0.log"
Oct 31 13:34:49.859014 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.858969    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t8wbk_bc47f7a4-649c-4529-a608-4479486a59d8/kube-rbac-proxy-main/0.log"
Oct 31 13:34:49.886749 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.886693    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t8wbk_bc47f7a4-649c-4529-a608-4479486a59d8/kube-rbac-proxy-self/0.log"
Oct 31 13:34:49.914061 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:49.914033    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-t8wbk_bc47f7a4-649c-4529-a608-4479486a59d8/openshift-state-metrics/0.log"
Oct 31 13:34:50.172534 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.172449    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r8ln9_a81cd320-aea5-4bbf-9a40-f83f814180b5/prometheus-operator/0.log"
Oct 31 13:34:50.200198 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.200175    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-r8ln9_a81cd320-aea5-4bbf-9a40-f83f814180b5/kube-rbac-proxy/0.log"
Oct 31 13:34:50.278202 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.278167    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dd9dd9b4d-zj579_85b46d9c-9428-4785-b432-0765ce8ea880/telemeter-client/0.log"
Oct 31 13:34:50.310853 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.310820    5787 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"]
Oct 31 13:34:50.311221 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.311209    5787 cpu_manager.go:414] "containerMap: removing container" podUID="74256a45-c6a4-42ef-b193-220f8b8e3224" containerName="revoke-keys"
Oct 31 13:34:50.311270 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.311222    5787 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74256a45-c6a4-42ef-b193-220f8b8e3224" containerName="revoke-keys"
Oct 31 13:34:50.311310 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.311291    5787 memory_manager.go:365] "RemoveStaleState removing state" podUID="74256a45-c6a4-42ef-b193-220f8b8e3224" containerName="revoke-keys"
Oct 31 13:34:50.313604 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.313584    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dd9dd9b4d-zj579_85b46d9c-9428-4785-b432-0765ce8ea880/reload/0.log"
Oct 31 13:34:50.314498 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.314482    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.319996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.319976    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xtmm\"/\"kube-root-ca.crt\""
Oct 31 13:34:50.324801 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.324755    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"]
Oct 31 13:34:50.326295 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.326274    5787 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8xtmm\"/\"default-dockercfg-c9924\""
Oct 31 13:34:50.353092 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.353058    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7dd9dd9b4d-zj579_85b46d9c-9428-4785-b432-0765ce8ea880/kube-rbac-proxy/0.log"
Oct 31 13:34:50.359843 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.359820    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-lib-modules\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.359996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.359848    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hdkr\" (UniqueName: \"kubernetes.io/projected/80d61d98-9030-4d2f-9004-b81e03fceaee-kube-api-access-7hdkr\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.359996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.359875    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-podres\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.360115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.360022    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-proc\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.360115 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.360074    5787 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-sys\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.388971 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.388943    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d6b4cbd64-96gzv_2bcc9522-04bc-46ec-873c-ac4d00b96ffb/thanos-query/0.log"
Oct 31 13:34:50.416910 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.416882    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d6b4cbd64-96gzv_2bcc9522-04bc-46ec-873c-ac4d00b96ffb/kube-rbac-proxy-web/0.log"
Oct 31 13:34:50.419566 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.419552    5787 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8xtmm\"/\"openshift-service-ca.crt\""
Oct 31 13:34:50.444996 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.444964    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d6b4cbd64-96gzv_2bcc9522-04bc-46ec-873c-ac4d00b96ffb/kube-rbac-proxy/0.log"
Oct 31 13:34:50.460831 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.460796    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-lib-modules\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.460831 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.460831    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hdkr\" (UniqueName: \"kubernetes.io/projected/80d61d98-9030-4d2f-9004-b81e03fceaee-kube-api-access-7hdkr\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.461046 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.460864    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-podres\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.461046 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.460925    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-proc\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.461046 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.460932    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-lib-modules\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.461046 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.460991    5787 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-sys\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.461046 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.461001    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-podres\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.461236 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.460998    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-proc\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.461236 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.461085    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/80d61d98-9030-4d2f-9004-b81e03fceaee-sys\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.473473 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.473440    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d6b4cbd64-96gzv_2bcc9522-04bc-46ec-873c-ac4d00b96ffb/prom-label-proxy/0.log"
Oct 31 13:34:50.474485 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.474463    5787 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hdkr\" (UniqueName: \"kubernetes.io/projected/80d61d98-9030-4d2f-9004-b81e03fceaee-kube-api-access-7hdkr\") pod \"perf-node-gather-daemonset-r66xn\" (UID: \"80d61d98-9030-4d2f-9004-b81e03fceaee\") " pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.500541 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.500514    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d6b4cbd64-96gzv_2bcc9522-04bc-46ec-873c-ac4d00b96ffb/kube-rbac-proxy-rules/0.log"
Oct 31 13:34:50.531419 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.531391    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d6b4cbd64-96gzv_2bcc9522-04bc-46ec-873c-ac4d00b96ffb/kube-rbac-proxy-metrics/0.log"
Oct 31 13:34:50.628906 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.628869    5787 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:50.906155 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:50.906128    5787 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"]
Oct 31 13:34:50.908868 ip-10-0-141-83 kubenswrapper[5787]: W1031 13:34:50.908833    5787 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod80d61d98_9030_4d2f_9004_b81e03fceaee.slice/crio-8df4405ea0d87cd2e0596a86b7a9e36ccd4088ef113968f11c8bb3823c048697 WatchSource:0}: Error finding container 8df4405ea0d87cd2e0596a86b7a9e36ccd4088ef113968f11c8bb3823c048697: Status 404 returned error can't find the container with id 8df4405ea0d87cd2e0596a86b7a9e36ccd4088ef113968f11c8bb3823c048697
Oct 31 13:34:51.671099 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:51.671064    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn" event={"ID":"80d61d98-9030-4d2f-9004-b81e03fceaee","Type":"ContainerStarted","Data":"0759cdf9905dc46d28a7757ab6dd6608a50d5dc41eef8634f64c2c4e241d007c"}
Oct 31 13:34:51.671588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:51.671122    5787 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn" event={"ID":"80d61d98-9030-4d2f-9004-b81e03fceaee","Type":"ContainerStarted","Data":"8df4405ea0d87cd2e0596a86b7a9e36ccd4088ef113968f11c8bb3823c048697"}
Oct 31 13:34:51.671588 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:51.671154    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:51.691690 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:51.691642    5787 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn" podStartSLOduration=1.691628152 podStartE2EDuration="1.691628152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:34:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:51.690256296 +0000 UTC m=+2071.106198135" watchObservedRunningTime="2026-10-31 13:34:51.691628152 +0000 UTC m=+2071.107569991"
Oct 31 13:34:52.950262 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:52.950226    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9d7869469-rl9wc_de46a2dc-38e7-493c-961b-b663ae7f7c27/console/0.log"
Oct 31 13:34:55.075564 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:55.075530    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wwg8z_3dc748e5-08b1-4ffa-900c-09730777cc8d/dns/0.log"
Oct 31 13:34:55.103180 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:55.103158    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wwg8z_3dc748e5-08b1-4ffa-900c-09730777cc8d/kube-rbac-proxy/0.log"
Oct 31 13:34:55.168327 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:55.168293    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ld8vz_448c900f-a9c5-4df1-b49e-e76269934da4/dns-node-resolver/0.log"
Oct 31 13:34:55.924547 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:55.924519    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lvtzl_e22c3df7-5789-4439-ba22-3a7d2b30c3fa/node-ca/0.log"
Oct 31 13:34:57.687723 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:57.687679    5787 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8xtmm/perf-node-gather-daemonset-r66xn"
Oct 31 13:34:58.233695 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:58.233660    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9bt9l_d192d1e3-c83f-4e9e-979f-684a6b98cc73/discovery/0.log"
Oct 31 13:34:58.391240 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:58.391213    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-72d7r_2b850a6e-358a-4baa-8730-9b569d126952/istio-proxy/0.log"
Oct 31 13:34:58.423486 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:58.423457    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8h4dd_80d56747-87ac-4de5-a195-7d50afcee740/istio-proxy/0.log"
Oct 31 13:34:58.563927 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:34:58.563832    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-rz8cm_d95c8381-0aff-448c-9422-d3693062872f/istio-proxy/0.log"
Oct 31 13:35:00.261284 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:00.261257    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2798k_7f24485c-3402-4cef-84eb-6ea5dd63be64/serve-healthcheck-canary/0.log"
Oct 31 13:35:01.486124 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:01.486086    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nkxs6_8b3fbffa-ba33-46db-9bdf-fcdf578955f6/kube-rbac-proxy/0.log"
Oct 31 13:35:01.515104 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:01.515074    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nkxs6_8b3fbffa-ba33-46db-9bdf-fcdf578955f6/exporter/0.log"
Oct 31 13:35:01.543809 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:01.543779    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nkxs6_8b3fbffa-ba33-46db-9bdf-fcdf578955f6/extractor/0.log"
Oct 31 13:35:02.131395 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:02.131343    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9bt9l_d192d1e3-c83f-4e9e-979f-684a6b98cc73/discovery/0.log"
Oct 31 13:35:02.261157 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:02.261126    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-72d7r_2b850a6e-358a-4baa-8730-9b569d126952/istio-proxy/0.log"
Oct 31 13:35:02.281328 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:02.281297    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8h4dd_80d56747-87ac-4de5-a195-7d50afcee740/istio-proxy/0.log"
Oct 31 13:35:02.365627 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:02.365601    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-rz8cm_d95c8381-0aff-448c-9422-d3693062872f/istio-proxy/0.log"
Oct 31 13:35:04.393733 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:04.393686    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-849556f89d-hf7p8_c8684b1b-03a2-4f50-946a-704fccf89e3e/manager/0.log"
Oct 31 13:35:04.423888 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:04.423858    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-849556f89d-hf7p8_c8684b1b-03a2-4f50-946a-704fccf89e3e/copy-manifests/0.log"
Oct 31 13:35:05.677049 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:05.677014    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-5c748f66b6-g58tt_ad206c3e-f20a-48bf-9290-7c681aaddfac/manager/0.log"
Oct 31 13:35:05.766428 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:05.766394    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-9bt9l_d192d1e3-c83f-4e9e-979f-684a6b98cc73/discovery/0.log"
Oct 31 13:35:05.917867 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:05.917838    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-72d7r_2b850a6e-358a-4baa-8730-9b569d126952/istio-proxy/0.log"
Oct 31 13:35:05.938491 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:05.938462    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-8h4dd_80d56747-87ac-4de5-a195-7d50afcee740/istio-proxy/0.log"
Oct 31 13:35:06.028469 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:06.028431    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f5667f86c-rz8cm_d95c8381-0aff-448c-9422-d3693062872f/istio-proxy/0.log"
Oct 31 13:35:06.440670 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:06.440643    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-9klng_dc065dda-a2b2-4b71-a8f8-8cc9298e40b7/manager/0.log"
Oct 31 13:35:06.756319 ip-10-0-141-83 kubenswrapper[5787]: I1031 13:35:06.756232    5787 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-5xbhb_5802a4b8-1826-4452-aab5-f8d2b2bc95a1/manager/0.log"