Sep 17 15:38:28.948750 ip-10-0-140-157 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 15:38:29.395426 ip-10-0-140-157 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:29.395426 ip-10-0-140-157 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 15:38:29.395426 ip-10-0-140-157 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:29.395426 ip-10-0-140-157 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 15:38:29.395426 ip-10-0-140-157 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 15:38:29.398094 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.398009    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 15:38:29.402459 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402443    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:29.402459 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402458    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:29.402459 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402474    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402478    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402483    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402486    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402489    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402492    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402497    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402500    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402504    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402506    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402509    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402518    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402521    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402524    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402527    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402530    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402533    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402537    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402540    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:29.402651 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402544    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402547    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402550    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402553    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402556    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402558    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402561    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402564    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402567    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402569    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402572    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402574    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402577    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402579    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402583    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402586    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402589    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402593    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402596    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402599    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:29.403144 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402602    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402606    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402609    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402611    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402614    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402616    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402619    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402622    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402625    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402627    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402630    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402632    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402635    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402638    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402640    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402643    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402646    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402648    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402651    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402653    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:29.403664 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402656    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402659    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402662    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402664    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402667    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402670    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402673    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402676    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402679    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402681    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402684    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402687    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402690    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402692    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402695    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402697    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402700    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402702    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402705    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402707    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:29.404153 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402710    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402712    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402715    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402717    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402720    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402723    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402727    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402729    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402732    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402736    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402739    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402742    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402745    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402747    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402751    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402754    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402756    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.402759    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403445    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:29.404652 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403452    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403455    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403458    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403473    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403476    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403479    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403482    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403484    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403487    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403490    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403492    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403495    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403498    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403500    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403503    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403505    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403508    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403510    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403513    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403515    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:29.405111 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403518    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403521    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403524    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403526    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403529    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403531    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403534    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403537    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403540    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403542    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403545    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403548    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403551    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403554    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403556    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403559    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403563    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403567    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403571    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:29.405656 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403574    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403577    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403579    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403582    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403585    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403587    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403590    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403592    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403595    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403598    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403601    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403603    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403606    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403610    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403614    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403617    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403619    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403622    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403625    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:29.406123 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403628    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403631    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403635    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403638    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403641    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403645    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403647    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403650    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403653    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403656    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403659    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403662    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403666    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403669    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403672    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403675    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403677    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403680    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403682    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403685    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:29.406610 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403687    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403690    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403693    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403695    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403698    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403700    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403703    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403706    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403710    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403712    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403715    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403718    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403721    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403723    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403727    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403730    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403733    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403736    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403738    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.403741    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:29.407095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404105    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404109    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404112    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404115    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404118    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404120    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404123    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404125    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404128    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404131    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404133    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404136    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404138    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404141    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404144    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404146    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404149    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404151    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404154    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404157    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:29.407609 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404159    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404162    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404165    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404168    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404171    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404173    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404176    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404179    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404181    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404184    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404186    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404189    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404191    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404194    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404196    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404199    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404201    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404204    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404207    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404209    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:29.408095 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404212    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404214    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404216    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404219    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404221    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404224    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404226    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404228    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404231    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404233    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404236    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404239    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404242    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404244    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404246    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404249    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404252    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404254    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404257    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404260    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:29.408602 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404262    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404264    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404269    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404272    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404275    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404278    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404281    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404283    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404286    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404289    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404292    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404295    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404299    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404304    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404307    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404310    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404312    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404315    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:29.409094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404317    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404320    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404322    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404325    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404327    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404331    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404333    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404336    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404338    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404341    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404344    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404346    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404349    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404351    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404354    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404356    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404359    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404361    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404364    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404366    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:29.409560 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.404368    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404439    2581 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404484    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404490    2581 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404494    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404500    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404503    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404507    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404511    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404514    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404517    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404521    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404524    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404527    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404530    2581 flags.go:64] FLAG: --cgroup-root=""
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404533    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404537    2581 flags.go:64] FLAG: --client-ca-file=""
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404540    2581 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404543    2581 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404547    2581 flags.go:64] FLAG: --cluster-domain=""
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404550    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404553    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404556    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404560    2581 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 15:38:29.410042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404563    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404566    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404570    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404573    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404576    2581 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404579    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404582    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404585    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404588    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404591    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404594    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404597    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404600    2581 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404603    2581 flags.go:64] FLAG: --enable-server="true"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404607    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404611    2581 flags.go:64] FLAG: --event-burst="100"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404615    2581 flags.go:64] FLAG: --event-qps="50"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404617    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404620    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404623    2581 flags.go:64] FLAG: --eviction-hard=""
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404631    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404634    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404637    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404640    2581 flags.go:64] FLAG: --eviction-soft=""
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404643    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 15:38:29.410664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404646    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404650    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404653    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404656    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404661    2581 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404664    2581 flags.go:64] FLAG: --feature-gates=""
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404668    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404671    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404674    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404678    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404681    2581 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404684    2581 flags.go:64] FLAG: --help="false"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404687    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404690    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404693    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404696    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404699    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404703    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404706    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404709    2581 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404712    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404715    2581 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404718    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404722    2581 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 15:38:29.411312 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404725    2581 flags.go:64] FLAG: --kube-reserved=""
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404728    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404730    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404733    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404736    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404739    2581 flags.go:64] FLAG: --lock-file=""
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404742    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404745    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404748    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404753    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404756    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404759    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404763    2581 flags.go:64] FLAG: --logging-format="text"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404766    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404770    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404773    2581 flags.go:64] FLAG: --manifest-url=""
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404775    2581 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404780    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404783    2581 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404787    2581 flags.go:64] FLAG: --max-pods="110"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404790    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404792    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404795    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404798    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404801    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 15:38:29.411932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404804    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404807    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404814    2581 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404817    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404820    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404823    2581 flags.go:64] FLAG: --pod-cidr=""
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404826    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404831    2581 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404834    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404837    2581 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404840    2581 flags.go:64] FLAG: --port="10250"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404843    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404845    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-018413f0c2f56b65e"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404849    2581 flags.go:64] FLAG: --qos-reserved=""
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404853    2581 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404857    2581 flags.go:64] FLAG: --register-node="true"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404859    2581 flags.go:64] FLAG: --register-with-taints=""
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404863    2581 flags.go:64] FLAG: --registry-burst="10"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404866    2581 flags.go:64] FLAG: --registry-qps="5"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404869    2581 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404873    2581 flags.go:64] FLAG: --reserved-memory=""
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404876    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404879    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404882    2581 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404885    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 15:38:29.412606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404889    2581 flags.go:64] FLAG: --runonce="false"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404892    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404894    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404897    2581 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404900    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404903    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404906    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404909    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404912    2581 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404915    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404918    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404921    2581 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404924    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404927    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404931    2581 flags.go:64] FLAG: --system-cgroups=""
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404934    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404939    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404942    2581 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404944    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404948    2581 flags.go:64] FLAG: --tls-min-version=""
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404952    2581 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404955    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404959    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404962    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404964    2581 flags.go:64] FLAG: --v="2"
Sep 17 15:38:29.413282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404968    2581 flags.go:64] FLAG: --version="false"
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404972    2581 flags.go:64] FLAG: --vmodule=""
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404977    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.404981    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405087    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405090    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405093    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405096    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405098    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405101    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405104    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405107    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405109    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405112    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405114    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405117    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405120    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405123    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405125    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405128    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:29.414003 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405131    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405133    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405136    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405139    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405142    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405144    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405146    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405149    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405153    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405156    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405159    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405161    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405164    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405166    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405170    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405173    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405175    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405178    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405180    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405183    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:29.414588 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405186    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405188    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405191    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405193    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405196    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405198    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405201    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405203    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405206    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405209    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405211    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405214    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405217    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405219    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405222    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405224    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405227    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405230    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405232    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405235    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:29.415164 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405238    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405241    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405243    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405246    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405249    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405251    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405257    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405260    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405262    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405265    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405269    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405272    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405275    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405278    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405281    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405283    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405285    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405288    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405292    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:29.415746 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405295    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405298    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405301    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405304    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405306    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405309    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405312    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405314    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405317    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405320    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405322    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405325    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405328    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405332    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405335    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405339    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405343    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405347    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405351    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:29.416274 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405355    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405358    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405360    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405363    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.405367    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.406275    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.414359    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.414373    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.414397    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.414403    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414486    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414492    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414495    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414498    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414501    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:29.416801 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414504    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414508    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414511    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414514    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414518    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414521    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414525    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414534    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414537    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414544    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414547    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414550    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414552    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414555    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414558    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414560    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414563    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414566    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414569    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:29.417183 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414571    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414574    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414577    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414580    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414582    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414585    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414588    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414591    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414594    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414596    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414599    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414602    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414604    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414607    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414610    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414613    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414615    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414618    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414622    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:29.417682 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414625    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414628    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414637    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414640    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414643    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414645    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414648    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414650    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414653    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414656    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414658    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414661    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414663    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414666    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414670    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414672    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414675    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414678    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414681    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414683    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:29.418157 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414686    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414689    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414691    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414694    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414697    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414699    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414702    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414704    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414707    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414709    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414712    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414715    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414717    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414720    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414722    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414731    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414734    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414736    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414739    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414742    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:29.418814 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414744    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414747    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414749    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414752    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414754    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414758    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414760    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414763    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414768    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414771    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414774    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414777    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414780    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414783    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414786    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:29.419310 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414789    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.414794    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414961    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414967    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414970    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414972    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414975    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414978    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414981    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414985    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414989    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414992    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.414995    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415004    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 15:38:29.419702 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415007    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415009    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415012    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415015    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415017    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415020    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415022    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415025    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415028    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415031    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415033    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415036    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415039    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415041    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415044    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415047    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415049    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415052    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415054    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415057    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 15:38:29.420073 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415059    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415062    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415064    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415067    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415069    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415072    2581 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415075    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415077    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415080    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415082    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415085    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415087    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415095    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415098    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415101    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415104    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415106    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415109    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415111    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415114    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 15:38:29.420570 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415117    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415120    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415122    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415125    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415128    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415130    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415133    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415135    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415137    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415140    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415142    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415145    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415147    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415150    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415152    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415155    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415157    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415160    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415164    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415168    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 15:38:29.421290 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415171    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415174    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415177    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415180    2581 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415182    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415191    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415195    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415198    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415201    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415203    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415206    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415208    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415211    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415213    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415216    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415219    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415221    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415224    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415226    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415229    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 15:38:29.422031 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415231    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415234    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415236    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415239    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415241    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415244    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:29.415246    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.415251    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.415357    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.417794    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.418793    2581 server.go:1019] "Starting client certificate rotation"
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.418892    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:38:29.422620 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.421517    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 15:38:29.425537 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.425521    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:38:29.426695 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.426681    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 15:38:29.435372 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.435355    2581 log.go:25] "Validated CRI v1 runtime API"
Sep 17 15:38:29.438477 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.438451    2581 log.go:25] "Validated CRI v1 image API"
Sep 17 15:38:29.442249 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.442233    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 15:38:29.446205 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.446185    2581 fs.go:135] Filesystem UUIDs: map[39fb45ea-4412-4795-8f6f-f80c133ac861:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9eba4c48-b89f-4657-b411-f1a9459e7271:/dev/nvme0n1p3]
Sep 17 15:38:29.446275 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.446203    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 15:38:29.452015 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.451903    2581 manager.go:216] Machine: {Timestamp:2026-09-17 15:38:29.450344412 +0000 UTC m=+0.373918973 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100254 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2bb9d503592dd2bd4f9b9af136e68e SystemUUID:ec2bb9d5-0359-2dd2-bd4f-9b9af136e68e BootID:ac08e5d3-c37f-4eb0-9e2e-f104b34a9454 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e0:5e:a2:05 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e0:5e:a2:05 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:fb:e1:18:6f:c1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 15:38:29.452015 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.452009    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 15:38:29.452120 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.452112    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 15:38:29.452381 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.452364    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:38:29.454773 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.454751    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 15:38:29.454909 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.454777    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-157.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 15:38:29.455584 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.455573    2581 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 15:38:29.455625 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.455585    2581 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 15:38:29.455625 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.455593    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:38:29.455625 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.455606    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 15:38:29.456831 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.456821    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 15:38:29.461375 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.461364    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:38:29.461505 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.461496    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 15:38:29.461535 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.461525    2581 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 15:38:29.461571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.461536    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 15:38:29.461571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.461548    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 15:38:29.461571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.461558    2581 kubelet.go:398] "Adding apiserver pod source"
Sep 17 15:38:29.461571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.461566    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 15:38:29.461910 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.461900    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:38:29.461945 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.461918    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 15:38:29.465103 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.465079    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 15:38:29.467513 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.467496    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 15:38:29.467601 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.467537    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 15:38:29.470352 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470338    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 15:38:29.470436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470358    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 15:38:29.470436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470368    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 15:38:29.470436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470375    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 15:38:29.470436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470387    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 15:38:29.470436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470395    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 15:38:29.470436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470413    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 15:38:29.470436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470421    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 15:38:29.470436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470431    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 15:38:29.470436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470439    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 15:38:29.470723 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470491    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 15:38:29.470723 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.470501    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 15:38:29.471443 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.471432    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 15:38:29.471511 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.471445    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 15:38:29.472509 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.472486    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 15:38:29.472509 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.472486    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 15:38:29.474315 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.474297    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-157.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 15:38:29.475293 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.475278    2581 server.go:1268] "Started kubelet"
Sep 17 15:38:29.475417 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.475360    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 15:38:29.475489 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.475362    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 15:38:29.475541 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.475512    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 15:38:29.476775 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.476758    2581 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 15:38:29.477591 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.477574    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 15:38:29.478250 ip-10-0-140-157 systemd[1]: Started Kubernetes Kubelet.
Sep 17 15:38:29.479750 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.479597    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 15:38:29.483008 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.481938    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-157.ec2.internal.18d625d7832c18d4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-157.ec2.internal,UID:ip-10-0-140-157.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-157.ec2.internal,},FirstTimestamp:2026-09-17 15:38:29.475252436 +0000 UTC m=+0.398826994,LastTimestamp:2026-09-17 15:38:29.475252436 +0000 UTC m=+0.398826994,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-157.ec2.internal,}"
Sep 17 15:38:29.484136 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.484119    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 15:38:29.484829 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.484816    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 15:38:29.484967 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.484946    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:29.485574 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485555    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 15:38:29.485574 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485556    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 15:38:29.485706 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485585    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 15:38:29.485706 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485598    2581 factory.go:55] Registering systemd factory
Sep 17 15:38:29.485706 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485610    2581 factory.go:223] Registration of the systemd container factory successfully
Sep 17 15:38:29.485706 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485698    2581 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 15:38:29.485706 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485709    2581 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 15:38:29.485920 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485821    2581 factory.go:153] Registering CRI-O factory
Sep 17 15:38:29.485920 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485837    2581 factory.go:223] Registration of the crio container factory successfully
Sep 17 15:38:29.485920 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485897    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 15:38:29.485920 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485914    2581 factory.go:103] Registering Raw factory
Sep 17 15:38:29.486093 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.485927    2581 manager.go:1196] Started watching for new ooms in manager
Sep 17 15:38:29.486407 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.486394    2581 manager.go:349] Starting recovery of all containers
Sep 17 15:38:29.486556 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.486532    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 15:38:29.486643 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.486617    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-157.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 15:38:29.497810 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.497792    2581 manager.go:354] Recovery completed
Sep 17 15:38:29.499849 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.499829    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kb64r"
Sep 17 15:38:29.502978 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.502967    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:29.505130 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.505114    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:29.505205 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.505145    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:29.505205 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.505159    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:29.505719 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.505703    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 15:38:29.505719 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.505715    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:29.505842 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.505717    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 15:38:29.505842 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.505767    2581 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 15:38:29.506371 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.506354    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:29.506454 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.506379    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:29.506454 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.506389    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:29.507230 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.507216    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kb64r"
Sep 17 15:38:29.508744 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.508731    2581 policy_none.go:49] "None policy: Start"
Sep 17 15:38:29.508803 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.508747    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 15:38:29.508803 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.508758    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 15:38:29.510682 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.510670    2581 policy_none.go:47] "Start"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.550297    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551081    2581 manager.go:337] "Starting Device Plugin manager"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.551119    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551132    2581 server.go:87] "Starting device plugin registration server"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551396    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551409    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551574    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551644    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551651    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551694    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551739    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.551761    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.551833    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.552260    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.552314    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:29.563078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.553994    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:29.651927 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.651890    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:29.652034 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.652006    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"]
Sep 17 15:38:29.652103 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.652094    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:29.654906 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.654889    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:29.655002 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.654922    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:29.655002 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.654937    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:29.655002 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.654965    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.655751 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.655738    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:29.655823 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.655764    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:29.655823 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.655779    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:29.658153 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.658140    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:29.658283 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.658261    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.658328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.658295    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:29.658861 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.658843    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:29.658922 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.658873    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:29.658922 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.658878    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:29.658922 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.658902    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:29.658922 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.658915    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:29.659033 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.658883    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:29.661116 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.661101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.661207 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.661129    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 15:38:29.661855 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.661840    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 15:38:29.661947 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.661871    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 15:38:29.661947 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.661886    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeHasSufficientPID"
Sep 17 15:38:29.662679 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.662665    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.662727 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.662684    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-157.ec2.internal\": node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:29.675077 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.675055    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:29.692987 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.692966    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-157.ec2.internal\" not found" node="ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.696996 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.696982    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-157.ec2.internal\" not found" node="ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.775128 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.775102    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:29.786031 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.786013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56f7058444d22e9cda05284d79371701-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"56f7058444d22e9cda05284d79371701\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.786114 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.786035    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5227f32ecce9c191f5f111ce617fad14-config\") pod \"kube-apiserver-proxy-ip-10-0-140-157.ec2.internal\" (UID: \"5227f32ecce9c191f5f111ce617fad14\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.786114 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.786053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56f7058444d22e9cda05284d79371701-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"56f7058444d22e9cda05284d79371701\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.875581 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.875558    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:29.886851 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.886831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56f7058444d22e9cda05284d79371701-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"56f7058444d22e9cda05284d79371701\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.886917 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.886864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56f7058444d22e9cda05284d79371701-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"56f7058444d22e9cda05284d79371701\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.886917 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.886893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5227f32ecce9c191f5f111ce617fad14-config\") pod \"kube-apiserver-proxy-ip-10-0-140-157.ec2.internal\" (UID: \"5227f32ecce9c191f5f111ce617fad14\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.887014 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.886912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/56f7058444d22e9cda05284d79371701-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"56f7058444d22e9cda05284d79371701\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.887014 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.886913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/56f7058444d22e9cda05284d79371701-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal\" (UID: \"56f7058444d22e9cda05284d79371701\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.887014 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.886977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5227f32ecce9c191f5f111ce617fad14-config\") pod \"kube-apiserver-proxy-ip-10-0-140-157.ec2.internal\" (UID: \"5227f32ecce9c191f5f111ce617fad14\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:29.976293 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:29.976237    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:29.995725 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:29.995705    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:30.001210 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.001014    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:30.077303 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.077273    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:30.177842 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.177814    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:30.277967 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.277932    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-157.ec2.internal\" not found"
Sep 17 15:38:30.331827 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.331802    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:30.385217 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.385192    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:30.403071 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.403054    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:38:30.405268 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.405254    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal"
Sep 17 15:38:30.416919 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.416902    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 15:38:30.418121 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.418104    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 15:38:30.418221 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.418206    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:38:30.418276 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.418257    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 15:38:30.464622 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.464591    2581 apiserver.go:52] "Watching apiserver"
Sep 17 15:38:30.478756 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.478734    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:30.481252 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.481236    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 15:38:30.484667 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.484643    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 15:38:30.485264 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.485242    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-7gbxk","openshift-image-registry/node-ca-48hkv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal","openshift-multus/multus-8b6mj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj","openshift-multus/multus-additional-cni-plugins-dcjdl","openshift-multus/network-metrics-daemon-lhp55","openshift-network-diagnostics/network-check-target-b6wrg","openshift-network-operator/iptables-alerter-rgzmz","openshift-ovn-kubernetes/ovnkube-node-486bt","kube-system/global-pull-secret-syncer-8lfn8","kube-system/konnectivity-agent-x2wr6","kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal","openshift-cluster-node-tuning-operator/tuned-s8zjw"]
Sep 17 15:38:30.487951 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.487934    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:30.488015 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.488002    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:30.490253 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.490237    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.491809 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.491757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:30.491809 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.491792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wn2\" (UniqueName: \"kubernetes.io/projected/4421c85c-5367-4632-b2c3-f91661885873-kube-api-access-p5wn2\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:30.492171 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.492156    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 15:38:30.492353 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.492340    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 15:38:30.492481 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.492469    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 15:38:30.492561 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.492537    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 15:38:30.492681 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.492659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.493091 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.493075    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vjv9q\""
Sep 17 15:38:30.494345 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.494324    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 15:38:30.494471 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.494358    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mrdhg\""
Sep 17 15:38:30.494471 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.494412    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 15:38:30.495064 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.495050    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.495706 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.495688    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 15:38:30.496719 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.496703    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 15:38:30.496801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.496766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 15:38:30.496896 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.496884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g5gmb\""
Sep 17 15:38:30.497028 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.497017    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 15:38:30.497502 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.497488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.499164 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.499150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w6tpk\""
Sep 17 15:38:30.499222 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.499210    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 15:38:30.499458 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.499443    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 15:38:30.499495 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.499469    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:38:30.499820 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.499803    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.501420 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.501406    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nkghk\""
Sep 17 15:38:30.501675 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.501660    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 15:38:30.501742 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.501708    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 15:38:30.502271 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.502255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.504059 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.504026    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 15:38:30.504168 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.504150    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-npq57\""
Sep 17 15:38:30.504236 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.504174    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 15:38:30.504236 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.504192    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 15:38:30.504792 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.504778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.506506 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.506486    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 15:38:30.506742 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.506728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2qkg\""
Sep 17 15:38:30.506808 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.506731    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 15:38:30.507094 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.507080    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 15:38:30.507170 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.507140    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 15:38:30.507234 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.507171    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:30.507280 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.507228    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:30.507320 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.507294    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 15:38:30.507320 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.507291    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 15:38:30.510058 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.510043    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:30.512113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.512098    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:30.512208 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.512159    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:30.512338 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.512321    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7krrp\""
Sep 17 15:38:30.512590 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.512559    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 15:33:29 +0000 UTC" deadline="2028-06-12 20:38:57.583000502 +0000 UTC"
Sep 17 15:38:30.512653 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.512591    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15221h0m27.070412623s"
Sep 17 15:38:30.513347 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.513330    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 15:38:30.514122 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.514007    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 15:38:30.514567 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.514532    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.516211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.516193    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 15:38:30.516722 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.516702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f95f2\""
Sep 17 15:38:30.516799 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.516768    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 15:38:30.522523 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.522508    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5dsxm"
Sep 17 15:38:30.529876 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.529858    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5dsxm"
Sep 17 15:38:30.586305 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.586288    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 15:38:30.592708 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592684    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-conf-dir\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.592786 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9f08147-9fea-4b2f-a2d7-59d0796228b2-system-cni-dir\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.592786 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-etc-selinux\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.592786 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-cni-netd\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.592786 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592774    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.592954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d033245e-d93b-4b38-b1c8-62351fd7cfc1-hosts-file\") pod \"node-resolver-7gbxk\" (UID: \"d033245e-d93b-4b38-b1c8-62351fd7cfc1\") " pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.592954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83930156-498b-4193-8569-cbc39a4658f9-agent-certs\") pod \"konnectivity-agent-x2wr6\" (UID: \"83930156-498b-4193-8569-cbc39a4658f9\") " pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:30.592954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.592954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-registration-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.592954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-kubelet\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.592954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-slash\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.592954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-etc-openvswitch\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.592954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkzq\" (UniqueName: \"kubernetes.io/projected/d033245e-d93b-4b38-b1c8-62351fd7cfc1-kube-api-access-mwkzq\") pod \"node-resolver-7gbxk\" (UID: \"d033245e-d93b-4b38-b1c8-62351fd7cfc1\") " pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.592954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-systemd\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-etc-kubernetes\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.592996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9f08147-9fea-4b2f-a2d7-59d0796228b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-run-openvswitch\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-dbus\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-system-cni-dir\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-sysconfig\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-sysctl-conf\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593130    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-sys\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/196dac60-e99e-407e-9d96-6d36b4febc02-tmp\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-var-lib-cni-bin\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593173    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-daemon-config\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593188    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9f08147-9fea-4b2f-a2d7-59d0796228b2-cnibin\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/015a1c41-25ba-4b7f-a281-626eeee00f40-cni-binary-copy\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43864a38-63f4-40b1-9728-0659647720c7-iptables-alerter-script\") pod \"iptables-alerter-rgzmz\" (UID: \"43864a38-63f4-40b1-9728-0659647720c7\") " pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.593328 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-var-lib-openvswitch\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-run-ovn\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593274    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-log-socket\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593293    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-cni-bin\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wn2\" (UniqueName: \"kubernetes.io/projected/4421c85c-5367-4632-b2c3-f91661885873-kube-api-access-p5wn2\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593339    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b5af873-2e95-452f-9eb6-936d70e56b0a-serviceca\") pod \"node-ca-48hkv\" (UID: \"6b5af873-2e95-452f-9eb6-936d70e56b0a\") " pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593379    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gn8x\" (UniqueName: \"kubernetes.io/projected/196dac60-e99e-407e-9d96-6d36b4febc02-kube-api-access-8gn8x\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-hostroot\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hxbj\" (UniqueName: \"kubernetes.io/projected/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-kube-api-access-4hxbj\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-systemd-units\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-node-log\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44355a48-2b12-4c88-8567-c926f8ba49f5-ovn-node-metrics-cert\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-kubernetes\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-host\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-cni-dir\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-var-lib-kubelet\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593667    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-var-lib-kubelet\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.593914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/196dac60-e99e-407e-9d96-6d36b4febc02-etc-tuned\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-sys-fs\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbqhh\" (UniqueName: \"kubernetes.io/projected/6b5af873-2e95-452f-9eb6-936d70e56b0a-kube-api-access-wbqhh\") pod \"node-ca-48hkv\" (UID: \"6b5af873-2e95-452f-9eb6-936d70e56b0a\") " pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593744    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83930156-498b-4193-8569-cbc39a4658f9-konnectivity-ca\") pod \"konnectivity-agent-x2wr6\" (UID: \"83930156-498b-4193-8569-cbc39a4658f9\") " pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-socket-dir-parent\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-run-systemd\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98vxk\" (UniqueName: \"kubernetes.io/projected/44355a48-2b12-4c88-8567-c926f8ba49f5-kube-api-access-98vxk\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593851    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-lib-modules\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghn5\" (UniqueName: \"kubernetes.io/projected/015a1c41-25ba-4b7f-a281-626eeee00f40-kube-api-access-5ghn5\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593888    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9f08147-9fea-4b2f-a2d7-59d0796228b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-run-netns\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44355a48-2b12-4c88-8567-c926f8ba49f5-ovnkube-config\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44355a48-2b12-4c88-8567-c926f8ba49f5-ovnkube-script-lib\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-cnibin\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-var-lib-cni-multus\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.594516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.593997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43864a38-63f4-40b1-9728-0659647720c7-host-slash\") pod \"iptables-alerter-rgzmz\" (UID: \"43864a38-63f4-40b1-9728-0659647720c7\") " pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9f08147-9fea-4b2f-a2d7-59d0796228b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-socket-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594053    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-device-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-sysctl-d\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzcts\" (UniqueName: \"kubernetes.io/projected/43864a38-63f4-40b1-9728-0659647720c7-kube-api-access-lzcts\") pod \"iptables-alerter-rgzmz\" (UID: \"43864a38-63f4-40b1-9728-0659647720c7\") " pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44355a48-2b12-4c88-8567-c926f8ba49f5-env-overrides\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d033245e-d93b-4b38-b1c8-62351fd7cfc1-tmp-dir\") pod \"node-resolver-7gbxk\" (UID: \"d033245e-d93b-4b38-b1c8-62351fd7cfc1\") " pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b5af873-2e95-452f-9eb6-936d70e56b0a-host\") pod \"node-ca-48hkv\" (UID: \"6b5af873-2e95-452f-9eb6-936d70e56b0a\") " pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-modprobe-d\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-run-netns\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594200    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9f08147-9fea-4b2f-a2d7-59d0796228b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-run-multus-certs\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prm6w\" (UniqueName: \"kubernetes.io/projected/e9f08147-9fea-4b2f-a2d7-59d0796228b2-kube-api-access-prm6w\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.594991 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-kubelet-config\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:30.595487 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-run\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.595487 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.594342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-run-k8s-cni-cncf-io\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.595487 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.594349    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:30.595487 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.594443    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs podName:4421c85c-5367-4632-b2c3-f91661885873 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:31.094426508 +0000 UTC m=+2.018001055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs") pod "network-metrics-daemon-lhp55" (UID: "4421c85c-5367-4632-b2c3-f91661885873") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:30.605723 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.605706    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 15:38:30.608450 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.608435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wn2\" (UniqueName: \"kubernetes.io/projected/4421c85c-5367-4632-b2c3-f91661885873-kube-api-access-p5wn2\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:30.638640 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.638617    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:30.695372 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-sys\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.695510 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/196dac60-e99e-407e-9d96-6d36b4febc02-tmp\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.695510 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-var-lib-cni-bin\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.695510 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-daemon-config\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.695510 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695441    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9f08147-9fea-4b2f-a2d7-59d0796228b2-cnibin\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.695510 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/015a1c41-25ba-4b7f-a281-626eeee00f40-cni-binary-copy\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695509    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43864a38-63f4-40b1-9728-0659647720c7-iptables-alerter-script\") pod \"iptables-alerter-rgzmz\" (UID: \"43864a38-63f4-40b1-9728-0659647720c7\") " pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-var-lib-openvswitch\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-run-ovn\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-sys\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-log-socket\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-cni-bin\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695613    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e9f08147-9fea-4b2f-a2d7-59d0796228b2-cnibin\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b5af873-2e95-452f-9eb6-936d70e56b0a-serviceca\") pod \"node-ca-48hkv\" (UID: \"6b5af873-2e95-452f-9eb6-936d70e56b0a\") " pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gn8x\" (UniqueName: \"kubernetes.io/projected/196dac60-e99e-407e-9d96-6d36b4febc02-kube-api-access-8gn8x\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-hostroot\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hxbj\" (UniqueName: \"kubernetes.io/projected/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-kube-api-access-4hxbj\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.695790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695789    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-systemd-units\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695812    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-node-log\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695839    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44355a48-2b12-4c88-8567-c926f8ba49f5-ovn-node-metrics-cert\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-run-ovn\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695868    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-kubernetes\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-var-lib-cni-bin\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695894    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-host\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-var-lib-openvswitch\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695919    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-cni-dir\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-cni-bin\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695956    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-var-lib-kubelet\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-var-lib-kubelet\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/196dac60-e99e-407e-9d96-6d36b4febc02-etc-tuned\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-sys-fs\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696083    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-var-lib-kubelet\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.695867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-log-socket\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-var-lib-kubelet\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-host\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.696387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-kubernetes\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbqhh\" (UniqueName: \"kubernetes.io/projected/6b5af873-2e95-452f-9eb6-936d70e56b0a-kube-api-access-wbqhh\") pod \"node-ca-48hkv\" (UID: \"6b5af873-2e95-452f-9eb6-936d70e56b0a\") " pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-node-log\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-cni-dir\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-systemd-units\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696228    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83930156-498b-4193-8569-cbc39a4658f9-konnectivity-ca\") pod \"konnectivity-agent-x2wr6\" (UID: \"83930156-498b-4193-8569-cbc39a4658f9\") " pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-hostroot\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-socket-dir-parent\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-run-systemd\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98vxk\" (UniqueName: \"kubernetes.io/projected/44355a48-2b12-4c88-8567-c926f8ba49f5-kube-api-access-98vxk\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-lib-modules\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghn5\" (UniqueName: \"kubernetes.io/projected/015a1c41-25ba-4b7f-a281-626eeee00f40-kube-api-access-5ghn5\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9f08147-9fea-4b2f-a2d7-59d0796228b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-run-netns\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44355a48-2b12-4c88-8567-c926f8ba49f5-ovnkube-config\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696528    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44355a48-2b12-4c88-8567-c926f8ba49f5-ovnkube-script-lib\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-cnibin\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.697318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-var-lib-cni-multus\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43864a38-63f4-40b1-9728-0659647720c7-host-slash\") pod \"iptables-alerter-rgzmz\" (UID: \"43864a38-63f4-40b1-9728-0659647720c7\") " pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9f08147-9fea-4b2f-a2d7-59d0796228b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696661    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6b5af873-2e95-452f-9eb6-936d70e56b0a-serviceca\") pod \"node-ca-48hkv\" (UID: \"6b5af873-2e95-452f-9eb6-936d70e56b0a\") " pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696669    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-socket-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-daemon-config\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696696    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-device-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-sysctl-d\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-run-netns\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzcts\" (UniqueName: \"kubernetes.io/projected/43864a38-63f4-40b1-9728-0659647720c7-kube-api-access-lzcts\") pod \"iptables-alerter-rgzmz\" (UID: \"43864a38-63f4-40b1-9728-0659647720c7\") " pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-sys-fs\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/43864a38-63f4-40b1-9728-0659647720c7-iptables-alerter-script\") pod \"iptables-alerter-rgzmz\" (UID: \"43864a38-63f4-40b1-9728-0659647720c7\") " pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696776    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44355a48-2b12-4c88-8567-c926f8ba49f5-env-overrides\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-socket-dir-parent\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696799    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d033245e-d93b-4b38-b1c8-62351fd7cfc1-tmp-dir\") pod \"node-resolver-7gbxk\" (UID: \"d033245e-d93b-4b38-b1c8-62351fd7cfc1\") " pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b5af873-2e95-452f-9eb6-936d70e56b0a-host\") pod \"node-ca-48hkv\" (UID: \"6b5af873-2e95-452f-9eb6-936d70e56b0a\") " pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-run-systemd\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-modprobe-d\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.698311 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.696868    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-run-netns\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9f08147-9fea-4b2f-a2d7-59d0796228b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.696904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/83930156-498b-4193-8569-cbc39a4658f9-konnectivity-ca\") pod \"konnectivity-agent-x2wr6\" (UID: \"83930156-498b-4193-8569-cbc39a4658f9\") " pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.696930    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret podName:0f9ba41d-7cfb-423d-acbb-8fee6d495fe7 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:31.196910763 +0000 UTC m=+2.120485321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret") pod "global-pull-secret-syncer-8lfn8" (UID: "0f9ba41d-7cfb-423d-acbb-8fee6d495fe7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-lib-modules\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697111    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-socket-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-cnibin\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697169    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-device-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-sysctl-d\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e9f08147-9fea-4b2f-a2d7-59d0796228b2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697523    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-var-lib-cni-multus\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/015a1c41-25ba-4b7f-a281-626eeee00f40-cni-binary-copy\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44355a48-2b12-4c88-8567-c926f8ba49f5-ovnkube-config\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e9f08147-9fea-4b2f-a2d7-59d0796228b2-cni-binary-copy\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697816    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-run-netns\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.697838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/43864a38-63f4-40b1-9728-0659647720c7-host-slash\") pod \"iptables-alerter-rgzmz\" (UID: \"43864a38-63f4-40b1-9728-0659647720c7\") " pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.699259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-modprobe-d\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698126    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d033245e-d93b-4b38-b1c8-62351fd7cfc1-tmp-dir\") pod \"node-resolver-7gbxk\" (UID: \"d033245e-d93b-4b38-b1c8-62351fd7cfc1\") " pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6b5af873-2e95-452f-9eb6-936d70e56b0a-host\") pod \"node-ca-48hkv\" (UID: \"6b5af873-2e95-452f-9eb6-936d70e56b0a\") " pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e9f08147-9fea-4b2f-a2d7-59d0796228b2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44355a48-2b12-4c88-8567-c926f8ba49f5-env-overrides\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44355a48-2b12-4c88-8567-c926f8ba49f5-ovnkube-script-lib\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-run-multus-certs\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prm6w\" (UniqueName: \"kubernetes.io/projected/e9f08147-9fea-4b2f-a2d7-59d0796228b2-kube-api-access-prm6w\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698836    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-kubelet-config\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-run\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698904    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-run-multus-certs\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-kubelet-config\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/196dac60-e99e-407e-9d96-6d36b4febc02-tmp\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-run\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-run-k8s-cni-cncf-io\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.698978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-host-run-k8s-cni-cncf-io\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.700192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699383    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-conf-dir\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699469    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-multus-conf-dir\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9f08147-9fea-4b2f-a2d7-59d0796228b2-system-cni-dir\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-etc-selinux\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699553    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e9f08147-9fea-4b2f-a2d7-59d0796228b2-system-cni-dir\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-etc-selinux\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699635    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-cni-netd\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-cni-netd\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d033245e-d93b-4b38-b1c8-62351fd7cfc1-hosts-file\") pod \"node-resolver-7gbxk\" (UID: \"d033245e-d93b-4b38-b1c8-62351fd7cfc1\") " pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83930156-498b-4193-8569-cbc39a4658f9-agent-certs\") pod \"konnectivity-agent-x2wr6\" (UID: \"83930156-498b-4193-8569-cbc39a4658f9\") " pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d033245e-d93b-4b38-b1c8-62351fd7cfc1-hosts-file\") pod \"node-resolver-7gbxk\" (UID: \"d033245e-d93b-4b38-b1c8-62351fd7cfc1\") " pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699845    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-registration-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699893    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-kubelet\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-slash\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-etc-openvswitch\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700002    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-kubelet\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkzq\" (UniqueName: \"kubernetes.io/projected/d033245e-d93b-4b38-b1c8-62351fd7cfc1-kube-api-access-mwkzq\") pod \"node-resolver-7gbxk\" (UID: \"d033245e-d93b-4b38-b1c8-62351fd7cfc1\") " pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-etc-openvswitch\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-systemd\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-host-slash\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-etc-kubernetes\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700110    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-systemd\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9f08147-9fea-4b2f-a2d7-59d0796228b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-etc-kubernetes\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-run-openvswitch\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-dbus\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.699934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-registration-dir\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700251    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-system-cni-dir\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-sysconfig\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.701905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-dbus\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:30.702517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700277    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44355a48-2b12-4c88-8567-c926f8ba49f5-run-openvswitch\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.702517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-sysconfig\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.702517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700408    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/196dac60-e99e-407e-9d96-6d36b4febc02-etc-tuned\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.702517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-sysctl-conf\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.702517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/196dac60-e99e-407e-9d96-6d36b4febc02-etc-sysctl-conf\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.702517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/015a1c41-25ba-4b7f-a281-626eeee00f40-system-cni-dir\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.702517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.700514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e9f08147-9fea-4b2f-a2d7-59d0796228b2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.702517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.701001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44355a48-2b12-4c88-8567-c926f8ba49f5-ovn-node-metrics-cert\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.702843 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.702678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/83930156-498b-4193-8569-cbc39a4658f9-agent-certs\") pod \"konnectivity-agent-x2wr6\" (UID: \"83930156-498b-4193-8569-cbc39a4658f9\") " pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:30.704435 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.704412    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbqhh\" (UniqueName: \"kubernetes.io/projected/6b5af873-2e95-452f-9eb6-936d70e56b0a-kube-api-access-wbqhh\") pod \"node-ca-48hkv\" (UID: \"6b5af873-2e95-452f-9eb6-936d70e56b0a\") " pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.704837 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.704810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hxbj\" (UniqueName: \"kubernetes.io/projected/de7e6aea-2ab1-4251-9b65-30fdd6e80c45-kube-api-access-4hxbj\") pod \"aws-ebs-csi-driver-node-mpcbj\" (UID: \"de7e6aea-2ab1-4251-9b65-30fdd6e80c45\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.705526 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.705499    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gn8x\" (UniqueName: \"kubernetes.io/projected/196dac60-e99e-407e-9d96-6d36b4febc02-kube-api-access-8gn8x\") pod \"tuned-s8zjw\" (UID: \"196dac60-e99e-407e-9d96-6d36b4febc02\") " pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.706499 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.706482    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:30.706692 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.706507    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:30.706692 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.706519    2581 projected.go:196] Error preparing data for projected volume kube-api-access-t9jxw for pod openshift-network-diagnostics/network-check-target-b6wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:30.706692 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.706625    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw podName:7fcd3586-396b-4b4d-945d-cc0f3853721b nodeName:}" failed. No retries permitted until 2026-09-17 15:38:31.206606895 +0000 UTC m=+2.130181457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t9jxw" (UniqueName: "kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw") pod "network-check-target-b6wrg" (UID: "7fcd3586-396b-4b4d-945d-cc0f3853721b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:30.707183 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.707155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghn5\" (UniqueName: \"kubernetes.io/projected/015a1c41-25ba-4b7f-a281-626eeee00f40-kube-api-access-5ghn5\") pod \"multus-8b6mj\" (UID: \"015a1c41-25ba-4b7f-a281-626eeee00f40\") " pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.707353 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.707327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzcts\" (UniqueName: \"kubernetes.io/projected/43864a38-63f4-40b1-9728-0659647720c7-kube-api-access-lzcts\") pod \"iptables-alerter-rgzmz\" (UID: \"43864a38-63f4-40b1-9728-0659647720c7\") " pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.708108 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.708081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98vxk\" (UniqueName: \"kubernetes.io/projected/44355a48-2b12-4c88-8567-c926f8ba49f5-kube-api-access-98vxk\") pod \"ovnkube-node-486bt\" (UID: \"44355a48-2b12-4c88-8567-c926f8ba49f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.708841 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.708821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prm6w\" (UniqueName: \"kubernetes.io/projected/e9f08147-9fea-4b2f-a2d7-59d0796228b2-kube-api-access-prm6w\") pod \"multus-additional-cni-plugins-dcjdl\" (UID: \"e9f08147-9fea-4b2f-a2d7-59d0796228b2\") " pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.708940 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.708913    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkzq\" (UniqueName: \"kubernetes.io/projected/d033245e-d93b-4b38-b1c8-62351fd7cfc1-kube-api-access-mwkzq\") pod \"node-resolver-7gbxk\" (UID: \"d033245e-d93b-4b38-b1c8-62351fd7cfc1\") " pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.813881 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.813851    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f7058444d22e9cda05284d79371701.slice/crio-a8b936bdf394c50ddf981c9fe43883ee7d2daaa6ebcc4f073230c9369b45950f WatchSource:0}: Error finding container a8b936bdf394c50ddf981c9fe43883ee7d2daaa6ebcc4f073230c9369b45950f: Status 404 returned error can't find the container with id a8b936bdf394c50ddf981c9fe43883ee7d2daaa6ebcc4f073230c9369b45950f
Sep 17 15:38:30.821139 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.821120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8b6mj"
Sep 17 15:38:30.821450 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.821435    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:38:30.828344 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.828293    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5227f32ecce9c191f5f111ce617fad14.slice/crio-d16c4cb6b8a591192c46a0265ff6a9c1b801897de7afe0af18d9ae5aa31aadb3 WatchSource:0}: Error finding container d16c4cb6b8a591192c46a0265ff6a9c1b801897de7afe0af18d9ae5aa31aadb3: Status 404 returned error can't find the container with id d16c4cb6b8a591192c46a0265ff6a9c1b801897de7afe0af18d9ae5aa31aadb3
Sep 17 15:38:30.831946 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.831917    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015a1c41_25ba_4b7f_a281_626eeee00f40.slice/crio-134a0918e8cf4c021b00ad6002524d9caf522db13ed4fee99cbd2ecffb07a9a0 WatchSource:0}: Error finding container 134a0918e8cf4c021b00ad6002524d9caf522db13ed4fee99cbd2ecffb07a9a0: Status 404 returned error can't find the container with id 134a0918e8cf4c021b00ad6002524d9caf522db13ed4fee99cbd2ecffb07a9a0
Sep 17 15:38:30.855851 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.855828    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dcjdl"
Sep 17 15:38:30.864871 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.864850    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9f08147_9fea_4b2f_a2d7_59d0796228b2.slice/crio-a15dda1b77ab0a536d840498446cceb071d69ec8501795e42ab51847bc33f18c WatchSource:0}: Error finding container a15dda1b77ab0a536d840498446cceb071d69ec8501795e42ab51847bc33f18c: Status 404 returned error can't find the container with id a15dda1b77ab0a536d840498446cceb071d69ec8501795e42ab51847bc33f18c
Sep 17 15:38:30.872856 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.872835    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj"
Sep 17 15:38:30.881085 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.881062    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7e6aea_2ab1_4251_9b65_30fdd6e80c45.slice/crio-3bbb3b4683ae3d0c2af4d2c5356677aca8ba8173efca9bcc32b7f1d4be0a9e33 WatchSource:0}: Error finding container 3bbb3b4683ae3d0c2af4d2c5356677aca8ba8173efca9bcc32b7f1d4be0a9e33: Status 404 returned error can't find the container with id 3bbb3b4683ae3d0c2af4d2c5356677aca8ba8173efca9bcc32b7f1d4be0a9e33
Sep 17 15:38:30.888204 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.888187    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rgzmz"
Sep 17 15:38:30.897115 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.897094    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43864a38_63f4_40b1_9728_0659647720c7.slice/crio-4de7ec583015229d406a9dff856d2eed2a1a3a7e87484ba67c9c7004ad23b3b3 WatchSource:0}: Error finding container 4de7ec583015229d406a9dff856d2eed2a1a3a7e87484ba67c9c7004ad23b3b3: Status 404 returned error can't find the container with id 4de7ec583015229d406a9dff856d2eed2a1a3a7e87484ba67c9c7004ad23b3b3
Sep 17 15:38:30.905775 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.905755    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7gbxk"
Sep 17 15:38:30.913478 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.913448    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd033245e_d93b_4b38_b1c8_62351fd7cfc1.slice/crio-e3f43ab391f1e0277d1f9fe8e9d434fbbbfc7199073002466028862250374caa WatchSource:0}: Error finding container e3f43ab391f1e0277d1f9fe8e9d434fbbbfc7199073002466028862250374caa: Status 404 returned error can't find the container with id e3f43ab391f1e0277d1f9fe8e9d434fbbbfc7199073002466028862250374caa
Sep 17 15:38:30.922467 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.922452    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-48hkv"
Sep 17 15:38:30.930396 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.930374    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b5af873_2e95_452f_9eb6_936d70e56b0a.slice/crio-c041a14d74bb0f4cd93a31a0e6e2dc21a58310f293e330c00b7b9fd113f80a51 WatchSource:0}: Error finding container c041a14d74bb0f4cd93a31a0e6e2dc21a58310f293e330c00b7b9fd113f80a51: Status 404 returned error can't find the container with id c041a14d74bb0f4cd93a31a0e6e2dc21a58310f293e330c00b7b9fd113f80a51
Sep 17 15:38:30.936577 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.936562    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:30.944155 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.944137    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44355a48_2b12_4c88_8567_c926f8ba49f5.slice/crio-363df4c668ae1fff9c4c3fb8ac31e8ce0dc745515b864ed91ba151db7c86bb7d WatchSource:0}: Error finding container 363df4c668ae1fff9c4c3fb8ac31e8ce0dc745515b864ed91ba151db7c86bb7d: Status 404 returned error can't find the container with id 363df4c668ae1fff9c4c3fb8ac31e8ce0dc745515b864ed91ba151db7c86bb7d
Sep 17 15:38:30.958687 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.958663    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:30.965751 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:30.965731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-s8zjw"
Sep 17 15:38:30.966554 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.966533    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83930156_498b_4193_8569_cbc39a4658f9.slice/crio-a33dcf2e7c5d9eef740224ce1a548b9a12cb90d3198a37d79139e8e89e6952cb WatchSource:0}: Error finding container a33dcf2e7c5d9eef740224ce1a548b9a12cb90d3198a37d79139e8e89e6952cb: Status 404 returned error can't find the container with id a33dcf2e7c5d9eef740224ce1a548b9a12cb90d3198a37d79139e8e89e6952cb
Sep 17 15:38:30.974320 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:38:30.974302    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod196dac60_e99e_407e_9d96_6d36b4febc02.slice/crio-10298a623d53192d85c6e8afc57bd187576dfedb3029167b9c0a0def968f773d WatchSource:0}: Error finding container 10298a623d53192d85c6e8afc57bd187576dfedb3029167b9c0a0def968f773d: Status 404 returned error can't find the container with id 10298a623d53192d85c6e8afc57bd187576dfedb3029167b9c0a0def968f773d
Sep 17 15:38:30.975838 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.975820    2581 kuberuntime_manager.go:1459] "Unhandled Error" err="container tuned start failed in pod tuned-s8zjw_openshift-cluster-node-tuning-operator(196dac60-e99e-407e-9d96-6d36b4febc02): ErrImagePull: pull QPS exceeded" logger="UnhandledError"
Sep 17 15:38:30.975897 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:30.975852    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-s8zjw" podUID="196dac60-e99e-407e-9d96-6d36b4febc02"
Sep 17 15:38:31.103691 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.103636    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:31.103795 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:31.103760    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:31.103836 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:31.103805    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs podName:4421c85c-5367-4632-b2c3-f91661885873 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:32.1037925 +0000 UTC m=+3.027367047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs") pod "network-metrics-daemon-lhp55" (UID: "4421c85c-5367-4632-b2c3-f91661885873") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:31.203993 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.203966    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:31.204086 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:31.204079    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:31.204135 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:31.204126    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret podName:0f9ba41d-7cfb-423d-acbb-8fee6d495fe7 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:32.204113969 +0000 UTC m=+3.127688516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret") pod "global-pull-secret-syncer-8lfn8" (UID: "0f9ba41d-7cfb-423d-acbb-8fee6d495fe7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:31.241299 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.241280    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:31.305266 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.305244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:31.305377 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:31.305364    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:31.305420 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:31.305382    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:31.305420 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:31.305392    2581 projected.go:196] Error preparing data for projected volume kube-api-access-t9jxw for pod openshift-network-diagnostics/network-check-target-b6wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:31.305506 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:31.305440    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw podName:7fcd3586-396b-4b4d-945d-cc0f3853721b nodeName:}" failed. No retries permitted until 2026-09-17 15:38:32.305424629 +0000 UTC m=+3.228999176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9jxw" (UniqueName: "kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw") pod "network-check-target-b6wrg" (UID: "7fcd3586-396b-4b4d-945d-cc0f3853721b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:31.533113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.533028    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:33:30 +0000 UTC" deadline="2028-03-24 01:00:36.047066648 +0000 UTC"
Sep 17 15:38:31.533113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.533058    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13281h22m4.514013232s"
Sep 17 15:38:31.558902 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.558841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8b6mj" event={"ID":"015a1c41-25ba-4b7f-a281-626eeee00f40","Type":"ContainerStarted","Data":"134a0918e8cf4c021b00ad6002524d9caf522db13ed4fee99cbd2ecffb07a9a0"}
Sep 17 15:38:31.559963 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.559932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s8zjw" event={"ID":"196dac60-e99e-407e-9d96-6d36b4febc02","Type":"ContainerStarted","Data":"10298a623d53192d85c6e8afc57bd187576dfedb3029167b9c0a0def968f773d"}
Sep 17 15:38:31.561345 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.561202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" event={"ID":"44355a48-2b12-4c88-8567-c926f8ba49f5","Type":"ContainerStarted","Data":"363df4c668ae1fff9c4c3fb8ac31e8ce0dc745515b864ed91ba151db7c86bb7d"}
Sep 17 15:38:31.561345 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:31.561304    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"tuned\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:667babb345aeb39d2681913fd91a5c163d48d4ab8a3ba0d3156502181ea1177a\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-cluster-node-tuning-operator/tuned-s8zjw" podUID="196dac60-e99e-407e-9d96-6d36b4febc02"
Sep 17 15:38:31.564648 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.564617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7gbxk" event={"ID":"d033245e-d93b-4b38-b1c8-62351fd7cfc1","Type":"ContainerStarted","Data":"e3f43ab391f1e0277d1f9fe8e9d434fbbbfc7199073002466028862250374caa"}
Sep 17 15:38:31.565729 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.565710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rgzmz" event={"ID":"43864a38-63f4-40b1-9728-0659647720c7","Type":"ContainerStarted","Data":"4de7ec583015229d406a9dff856d2eed2a1a3a7e87484ba67c9c7004ad23b3b3"}
Sep 17 15:38:31.566726 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.566697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj" event={"ID":"de7e6aea-2ab1-4251-9b65-30fdd6e80c45","Type":"ContainerStarted","Data":"3bbb3b4683ae3d0c2af4d2c5356677aca8ba8173efca9bcc32b7f1d4be0a9e33"}
Sep 17 15:38:31.567847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.567825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcjdl" event={"ID":"e9f08147-9fea-4b2f-a2d7-59d0796228b2","Type":"ContainerStarted","Data":"a15dda1b77ab0a536d840498446cceb071d69ec8501795e42ab51847bc33f18c"}
Sep 17 15:38:31.568927 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.568908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal" event={"ID":"5227f32ecce9c191f5f111ce617fad14","Type":"ContainerStarted","Data":"d16c4cb6b8a591192c46a0265ff6a9c1b801897de7afe0af18d9ae5aa31aadb3"}
Sep 17 15:38:31.570037 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.570018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal" event={"ID":"56f7058444d22e9cda05284d79371701","Type":"ContainerStarted","Data":"a8b936bdf394c50ddf981c9fe43883ee7d2daaa6ebcc4f073230c9369b45950f"}
Sep 17 15:38:31.571142 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.571117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x2wr6" event={"ID":"83930156-498b-4193-8569-cbc39a4658f9","Type":"ContainerStarted","Data":"a33dcf2e7c5d9eef740224ce1a548b9a12cb90d3198a37d79139e8e89e6952cb"}
Sep 17 15:38:31.572209 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:31.572191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-48hkv" event={"ID":"6b5af873-2e95-452f-9eb6-936d70e56b0a","Type":"ContainerStarted","Data":"c041a14d74bb0f4cd93a31a0e6e2dc21a58310f293e330c00b7b9fd113f80a51"}
Sep 17 15:38:32.112009 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:32.111965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:32.112224 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.112179    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:32.112296 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.112251    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs podName:4421c85c-5367-4632-b2c3-f91661885873 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:34.112228733 +0000 UTC m=+5.035803298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs") pod "network-metrics-daemon-lhp55" (UID: "4421c85c-5367-4632-b2c3-f91661885873") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:32.179506 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:32.179472    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 15:38:32.212838 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:32.212803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:32.213026 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.212956    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:32.213026 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.213020    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret podName:0f9ba41d-7cfb-423d-acbb-8fee6d495fe7 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:34.213002185 +0000 UTC m=+5.136576734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret") pod "global-pull-secret-syncer-8lfn8" (UID: "0f9ba41d-7cfb-423d-acbb-8fee6d495fe7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:32.313246 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:32.313210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:32.313451 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.313376    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:32.313451 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.313399    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:32.313451 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.313412    2581 projected.go:196] Error preparing data for projected volume kube-api-access-t9jxw for pod openshift-network-diagnostics/network-check-target-b6wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:32.313648 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.313476    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw podName:7fcd3586-396b-4b4d-945d-cc0f3853721b nodeName:}" failed. No retries permitted until 2026-09-17 15:38:34.313456991 +0000 UTC m=+5.237031540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9jxw" (UniqueName: "kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw") pod "network-check-target-b6wrg" (UID: "7fcd3586-396b-4b4d-945d-cc0f3853721b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:32.534184 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:32.534067    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 15:33:30 +0000 UTC" deadline="2028-04-10 08:27:32.157363676 +0000 UTC"
Sep 17 15:38:32.534184 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:32.534131    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13696h48m59.623237341s"
Sep 17 15:38:32.552600 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:32.552565    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:32.552733 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.552712    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:32.553126 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:32.553108    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:32.553253 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.553213    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:32.553253 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:32.553276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:32.553442 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:32.553360    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:34.130426 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:34.130381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:34.131018 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.130611    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:34.131018 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.130672    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs podName:4421c85c-5367-4632-b2c3-f91661885873 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:38.130655116 +0000 UTC m=+9.054229668 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs") pod "network-metrics-daemon-lhp55" (UID: "4421c85c-5367-4632-b2c3-f91661885873") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:34.231992 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:34.231364    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:34.231992 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.231594    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:34.231992 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.231668    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret podName:0f9ba41d-7cfb-423d-acbb-8fee6d495fe7 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:38.231650174 +0000 UTC m=+9.155224733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret") pod "global-pull-secret-syncer-8lfn8" (UID: "0f9ba41d-7cfb-423d-acbb-8fee6d495fe7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:34.332432 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:34.332397    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:34.332673 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.332651    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:34.332769 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.332680    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:34.332769 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.332693    2581 projected.go:196] Error preparing data for projected volume kube-api-access-t9jxw for pod openshift-network-diagnostics/network-check-target-b6wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:34.332769 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.332749    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw podName:7fcd3586-396b-4b4d-945d-cc0f3853721b nodeName:}" failed. No retries permitted until 2026-09-17 15:38:38.332731487 +0000 UTC m=+9.256306036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9jxw" (UniqueName: "kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw") pod "network-check-target-b6wrg" (UID: "7fcd3586-396b-4b4d-945d-cc0f3853721b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:34.552759 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:34.552733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:34.552901 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.552884    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:34.552976 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:34.552916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:34.553029 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.553006    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:34.553077 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:34.553045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:34.553129 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:34.553117    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:36.552387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:36.552356    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:36.552831 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:36.552487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:36.552831 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:36.552493    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:36.552831 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:36.552505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:36.552831 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:36.552574    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:36.552831 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:36.552647    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:38.164002 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:38.163967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:38.164445 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.164111    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:38.164445 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.164173    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs podName:4421c85c-5367-4632-b2c3-f91661885873 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:46.16415459 +0000 UTC m=+17.087729161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs") pod "network-metrics-daemon-lhp55" (UID: "4421c85c-5367-4632-b2c3-f91661885873") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:38.264735 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:38.264703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:38.264901 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.264874    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:38.264963 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.264933    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret podName:0f9ba41d-7cfb-423d-acbb-8fee6d495fe7 nodeName:}" failed. No retries permitted until 2026-09-17 15:38:46.264916941 +0000 UTC m=+17.188491492 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret") pod "global-pull-secret-syncer-8lfn8" (UID: "0f9ba41d-7cfb-423d-acbb-8fee6d495fe7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:38.365320 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:38.365235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:38.365498 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.365409    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:38.365498 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.365433    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:38.365498 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.365446    2581 projected.go:196] Error preparing data for projected volume kube-api-access-t9jxw for pod openshift-network-diagnostics/network-check-target-b6wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:38.365628 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.365513    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw podName:7fcd3586-396b-4b4d-945d-cc0f3853721b nodeName:}" failed. No retries permitted until 2026-09-17 15:38:46.365498484 +0000 UTC m=+17.289073030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9jxw" (UniqueName: "kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw") pod "network-check-target-b6wrg" (UID: "7fcd3586-396b-4b4d-945d-cc0f3853721b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:38.552954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:38.552885    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:38.553143 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:38.552884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:38.553143 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.553065    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:38.553143 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.553082    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:38.553143 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:38.552916    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:38.553352 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:38.553164    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:40.552427 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:40.552388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:40.552427 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:40.552403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:40.552969 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:40.552388    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:40.552969 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:40.552560    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:40.552969 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:40.552656    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:40.552969 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:40.552815    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:42.552632 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:42.552599    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:42.552632 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:42.552621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:42.553150 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:42.552621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:42.553150 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:42.552731    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:42.553150 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:42.552803    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:42.553150 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:42.552898    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:44.552990 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:44.552954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:44.553432 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:44.553066    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:44.553432 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:44.553083    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:44.553432 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:44.553105    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:44.553432 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:44.553207    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:44.553432 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:44.553295    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:46.218437 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:46.218398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:46.218909 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.218563    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:46.218909 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.218620    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs podName:4421c85c-5367-4632-b2c3-f91661885873 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:02.218601628 +0000 UTC m=+33.142176189 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs") pod "network-metrics-daemon-lhp55" (UID: "4421c85c-5367-4632-b2c3-f91661885873") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:38:46.319416 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:46.319377    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:46.319601 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.319542    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:46.319657 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.319615    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret podName:0f9ba41d-7cfb-423d-acbb-8fee6d495fe7 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:02.31959379 +0000 UTC m=+33.243168337 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret") pod "global-pull-secret-syncer-8lfn8" (UID: "0f9ba41d-7cfb-423d-acbb-8fee6d495fe7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:38:46.420733 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:46.420704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:46.420910 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.420879    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:38:46.420910 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.420899    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:38:46.420910 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.420909    2581 projected.go:196] Error preparing data for projected volume kube-api-access-t9jxw for pod openshift-network-diagnostics/network-check-target-b6wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:46.421067 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.420963    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw podName:7fcd3586-396b-4b4d-945d-cc0f3853721b nodeName:}" failed. No retries permitted until 2026-09-17 15:39:02.420946901 +0000 UTC m=+33.344521451 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9jxw" (UniqueName: "kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw") pod "network-check-target-b6wrg" (UID: "7fcd3586-396b-4b4d-945d-cc0f3853721b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:38:46.552797 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:46.552768    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:46.553016 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:46.552803    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:46.553016 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.552881    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:46.553016 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:46.552765    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:46.553167 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.553022    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:46.553167 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:46.553104    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:48.553076 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.552853    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:48.553765 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.552895    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:48.553765 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:48.553174    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:48.553765 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.552935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:48.553765 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:48.553251    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:48.553765 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:48.553351    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:48.608269 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.608174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-x2wr6" event={"ID":"83930156-498b-4193-8569-cbc39a4658f9","Type":"ContainerStarted","Data":"06006fa832d36ef1eec75bb1b18fd348f8670019b606dedab8f9a6bcfdb086b3"}
Sep 17 15:38:48.609915 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.609891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-48hkv" event={"ID":"6b5af873-2e95-452f-9eb6-936d70e56b0a","Type":"ContainerStarted","Data":"15b044b5706c291aae37dc14ea5c433781a582c9cd0680bb447fdcc10c2e9750"}
Sep 17 15:38:48.611637 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.611603    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8b6mj" event={"ID":"015a1c41-25ba-4b7f-a281-626eeee00f40","Type":"ContainerStarted","Data":"dc756496bb5434ca4d7253ee89727f7ef9499b75f24962821ad5cee0ce03f24a"}
Sep 17 15:38:48.615179 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.615158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" event={"ID":"44355a48-2b12-4c88-8567-c926f8ba49f5","Type":"ContainerStarted","Data":"034ff2a59fc9a4be459e67955269b2eca6fe07915f03a38d0366984a6921e6c6"}
Sep 17 15:38:48.615272 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.615231    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" event={"ID":"44355a48-2b12-4c88-8567-c926f8ba49f5","Type":"ContainerStarted","Data":"839c8be4216eabb6dcc3e715626f15bd6845cc3c05bde24440be0c6d878d31a7"}
Sep 17 15:38:48.615342 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.615286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" event={"ID":"44355a48-2b12-4c88-8567-c926f8ba49f5","Type":"ContainerStarted","Data":"60b2aadb18c03881904f698e62a07d23e10fcc44d9f394f1022d007655c483a9"}
Sep 17 15:38:48.615342 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.615334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" event={"ID":"44355a48-2b12-4c88-8567-c926f8ba49f5","Type":"ContainerStarted","Data":"dc6a588889c9a606f0005bd75a45e1f340fe9b82466c67b854e1e11e7827518c"}
Sep 17 15:38:48.615441 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.615381    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" event={"ID":"44355a48-2b12-4c88-8567-c926f8ba49f5","Type":"ContainerStarted","Data":"ac90415aeb50ed463fa03826338c8d16ac347a426f6496b36cf3761aa114a1d8"}
Sep 17 15:38:48.615441 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.615424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" event={"ID":"44355a48-2b12-4c88-8567-c926f8ba49f5","Type":"ContainerStarted","Data":"1faf9da9efafcb8896b1e1679369cc6d0e5b074630cf65b3a842901fe42b1e3e"}
Sep 17 15:38:48.616755 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.616725    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7gbxk" event={"ID":"d033245e-d93b-4b38-b1c8-62351fd7cfc1","Type":"ContainerStarted","Data":"c3bee7ed2facb3e0be6dc9c37b0d1eb00e9a227272d0f0327c12e55665912152"}
Sep 17 15:38:48.618292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.618272    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj" event={"ID":"de7e6aea-2ab1-4251-9b65-30fdd6e80c45","Type":"ContainerStarted","Data":"58e97fad3b538d449c8f790791368cea7afefe767b5ee4d6703fb532de1a4b88"}
Sep 17 15:38:48.619979 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.619957    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9f08147-9fea-4b2f-a2d7-59d0796228b2" containerID="93c6fcee3cd9e3fc48fdf80f283508ed2ebd7674516f1197ff147a4adcf3c1b1" exitCode=0
Sep 17 15:38:48.620057 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.620035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcjdl" event={"ID":"e9f08147-9fea-4b2f-a2d7-59d0796228b2","Type":"ContainerDied","Data":"93c6fcee3cd9e3fc48fdf80f283508ed2ebd7674516f1197ff147a4adcf3c1b1"}
Sep 17 15:38:48.621778 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.621752    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal" event={"ID":"5227f32ecce9c191f5f111ce617fad14","Type":"ContainerStarted","Data":"8f672e82485b69533b2eb1288cbfb67c8fc23e11d5d53a4d39073c29dce3c9b0"}
Sep 17 15:38:48.623294 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.623267    2581 generic.go:356] "Generic (PLEG): container finished" podID="56f7058444d22e9cda05284d79371701" containerID="8ff8b640adc6a957fc02aff7d4d2caa8bec3598974ac5b8d4677a3859f89cc4d" exitCode=0
Sep 17 15:38:48.623374 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.623298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal" event={"ID":"56f7058444d22e9cda05284d79371701","Type":"ContainerDied","Data":"8ff8b640adc6a957fc02aff7d4d2caa8bec3598974ac5b8d4677a3859f89cc4d"}
Sep 17 15:38:48.639503 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.639438    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8b6mj" podStartSLOduration=2.5162292109999997 podStartE2EDuration="19.639425295s" totalImagesPullingTime="17.123196084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:48.63936434 +0000 UTC m=+19.562938908" watchObservedRunningTime="2026-09-17 15:38:48.639425295 +0000 UTC m=+19.562999863"
Sep 17 15:38:48.640067 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.640030    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-x2wr6" podStartSLOduration=1.8631545109999998 podStartE2EDuration="18.64001748s" totalImagesPullingTime="16.776862969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:48.62488524 +0000 UTC m=+19.548459810" watchObservedRunningTime="2026-09-17 15:38:48.64001748 +0000 UTC m=+19.563592048"
Sep 17 15:38:48.670798 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.670492    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7gbxk" podStartSLOduration=1.8354216920000002 podStartE2EDuration="18.670438091s" totalImagesPullingTime="16.835016399s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:48.669705124 +0000 UTC m=+19.593279691" watchObservedRunningTime="2026-09-17 15:38:48.670438091 +0000 UTC m=+19.594012659"
Sep 17 15:38:48.697481 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.697421    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-157.ec2.internal" podStartSLOduration=18.697403966 podStartE2EDuration="18.697403966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:48.697358583 +0000 UTC m=+19.620933162" watchObservedRunningTime="2026-09-17 15:38:48.697403966 +0000 UTC m=+19.620978537"
Sep 17 15:38:48.697681 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:48.697650    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-48hkv" podStartSLOduration=1.9054529599999999 podStartE2EDuration="18.697639483s" totalImagesPullingTime="16.792186523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:48.685082794 +0000 UTC m=+19.608657363" watchObservedRunningTime="2026-09-17 15:38:48.697639483 +0000 UTC m=+19.621214053"
Sep 17 15:38:49.486443 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:49.486415    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 15:38:49.561649 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:49.561543    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T15:38:49.486439855Z","UUID":"673912d5-e706-4ba1-ab19-91699ae1c0b1","Handler":null,"Name":"","Endpoint":""}
Sep 17 15:38:49.563986 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:49.563964    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 15:38:49.564113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:49.564001    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 15:38:49.627587 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:49.627483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rgzmz" event={"ID":"43864a38-63f4-40b1-9728-0659647720c7","Type":"ContainerStarted","Data":"c4e51322df14d2bafaf407d6607a909519b8ee01a37bec39d0b4873292e9e841"}
Sep 17 15:38:49.630031 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:49.629996    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj" event={"ID":"de7e6aea-2ab1-4251-9b65-30fdd6e80c45","Type":"ContainerStarted","Data":"0ad462b04f0ef3961d9175e17774be243a72b56e5580ed6959f862c2f44bd06a"}
Sep 17 15:38:49.632838 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:49.632816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal" event={"ID":"56f7058444d22e9cda05284d79371701","Type":"ContainerStarted","Data":"f5d03127074debe8a95d80a2017a552dd4d249d46a4492b29469e02e40b9673a"}
Sep 17 15:38:49.642071 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:49.642025    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rgzmz" podStartSLOduration=2.816639245 podStartE2EDuration="19.642007237s" totalImagesPullingTime="16.825367992s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:49.641508777 +0000 UTC m=+20.565083345" watchObservedRunningTime="2026-09-17 15:38:49.642007237 +0000 UTC m=+20.565581806"
Sep 17 15:38:49.655371 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:49.655330    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-157.ec2.internal" podStartSLOduration=19.655314343 podStartE2EDuration="19.655314343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:49.655009441 +0000 UTC m=+20.578584013" watchObservedRunningTime="2026-09-17 15:38:49.655314343 +0000 UTC m=+20.578888911"
Sep 17 15:38:50.552352 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:50.552316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:50.552352 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:50.552340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:50.552578 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:50.552329    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:50.552578 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:50.552449    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:50.552578 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:50.552565    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:50.552709 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:50.552654    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:51.801082 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:51.801050    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:51.801957 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:51.801930    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:52.552517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:52.552488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:52.552707 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:52.552584    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:52.552707 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:52.552597    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:52.552707 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:52.552489    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:52.552707 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:52.552679    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:52.552840 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:52.552754    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:53.642536 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:53.642282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-s8zjw" event={"ID":"196dac60-e99e-407e-9d96-6d36b4febc02","Type":"ContainerStarted","Data":"90ed4c75c0da5ec887ea5ae37c83bea90f95b594ead17ec36fef8db3cc51ab08"}
Sep 17 15:38:53.646234 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:53.646202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" event={"ID":"44355a48-2b12-4c88-8567-c926f8ba49f5","Type":"ContainerStarted","Data":"9275dd83c1e518c911e581c031ed1a13a31c184e173b406b5d7ebbc1b30d82b1"}
Sep 17 15:38:53.648783 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:53.648751    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj" event={"ID":"de7e6aea-2ab1-4251-9b65-30fdd6e80c45","Type":"ContainerStarted","Data":"42377146d7f840e1b1bff59f1f8636c62dac17806f4088db6146441a3f91fed0"}
Sep 17 15:38:53.659232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:53.659187    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-s8zjw" podStartSLOduration=1.7048883510000001 podStartE2EDuration="23.659173491s" totalImagesPullingTime="21.95428514s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-09-17 15:38:53.658624028 +0000 UTC m=+24.582198595" watchObservedRunningTime="2026-09-17 15:38:53.659173491 +0000 UTC m=+24.582748059"
Sep 17 15:38:53.673587 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:53.673539    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mpcbj" podStartSLOduration=2.315674542 podStartE2EDuration="23.673526254s" totalImagesPullingTime="21.357851712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:53.673270442 +0000 UTC m=+24.596845009" watchObservedRunningTime="2026-09-17 15:38:53.673526254 +0000 UTC m=+24.597100821"
Sep 17 15:38:54.552651 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:54.552615    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:54.552651 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:54.552653    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:54.552903 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:54.552751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:54.552903 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:54.552764    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:54.552903 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:54.552835    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:54.553041 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:54.552930    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:55.655742 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:55.655658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" event={"ID":"44355a48-2b12-4c88-8567-c926f8ba49f5","Type":"ContainerStarted","Data":"275ade6e1329faa65ea44df4af6633eac1b68405a59e00c77d50c19f57236ac6"}
Sep 17 15:38:55.656165 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:55.655930    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:55.656165 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:55.655996    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:55.657576 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:55.657552    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9f08147-9fea-4b2f-a2d7-59d0796228b2" containerID="fb650ae6a67861209559e1f5ed152da8ff1bbeb05fc39f6ea204978bc59970c9" exitCode=0
Sep 17 15:38:55.657701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:55.657589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcjdl" event={"ID":"e9f08147-9fea-4b2f-a2d7-59d0796228b2","Type":"ContainerDied","Data":"fb650ae6a67861209559e1f5ed152da8ff1bbeb05fc39f6ea204978bc59970c9"}
Sep 17 15:38:55.671546 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:55.671527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:55.679673 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:55.679639    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-486bt" podStartSLOduration=8.619155148 podStartE2EDuration="25.679627928s" totalImagesPullingTime="17.06047278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:38:55.679548774 +0000 UTC m=+26.603123341" watchObservedRunningTime="2026-09-17 15:38:55.679627928 +0000 UTC m=+26.603202494"
Sep 17 15:38:56.552994 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:56.552963    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:56.553275 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:56.552971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:56.553275 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:56.553095    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:56.553275 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:56.552983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:56.553275 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:56.553195    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:56.553511 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:56.553325    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:56.660902 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:56.660875    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:56.676043 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:56.676017    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:38:57.003841 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:57.003793    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8lfn8"]
Sep 17 15:38:57.003967 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:57.003949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:57.004239 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:57.004063    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:57.009357 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:57.009334    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhp55"]
Sep 17 15:38:57.009489 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:57.009457    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:57.009612 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:57.009579    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:57.010130 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:57.010095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b6wrg"]
Sep 17 15:38:57.010253 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:57.010190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:57.010344 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:57.010283    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:57.664871 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:57.664836    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9f08147-9fea-4b2f-a2d7-59d0796228b2" containerID="21001b888b58033475ae96a4d1e83740d0f701b1adfb0341f3f0cb1e2fb70fc5" exitCode=0
Sep 17 15:38:57.665520 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:57.664929    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcjdl" event={"ID":"e9f08147-9fea-4b2f-a2d7-59d0796228b2","Type":"ContainerDied","Data":"21001b888b58033475ae96a4d1e83740d0f701b1adfb0341f3f0cb1e2fb70fc5"}
Sep 17 15:38:58.193203 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:58.192993    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:58.193371 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:58.193292    2581 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 15:38:58.193638 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:58.193621    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-x2wr6"
Sep 17 15:38:58.552584 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:58.552564    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:38:58.552689 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:58.552569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:38:58.552689 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:58.552657    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:38:58.552689 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:58.552679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:38:58.552847 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:58.552758    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:38:58.552847 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:38:58.552837    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:38:58.669137 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:58.669109    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9f08147-9fea-4b2f-a2d7-59d0796228b2" containerID="68e3e33a664e4876e72876e99a78fe6836d22377420bcf976f5495812528497e" exitCode=0
Sep 17 15:38:58.669547 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:38:58.669203    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcjdl" event={"ID":"e9f08147-9fea-4b2f-a2d7-59d0796228b2","Type":"ContainerDied","Data":"68e3e33a664e4876e72876e99a78fe6836d22377420bcf976f5495812528497e"}
Sep 17 15:39:00.552458 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:00.552423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:39:00.552883 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:00.552434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:39:00.552883 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:00.552562    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b6wrg" podUID="7fcd3586-396b-4b4d-945d-cc0f3853721b"
Sep 17 15:39:00.552883 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:00.552662    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8lfn8" podUID="0f9ba41d-7cfb-423d-acbb-8fee6d495fe7"
Sep 17 15:39:00.552883 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:00.552444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:39:00.552883 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:00.552780    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lhp55" podUID="4421c85c-5367-4632-b2c3-f91661885873"
Sep 17 15:39:01.844352 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.844328    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-157.ec2.internal" event="NodeReady"
Sep 17 15:39:01.844731 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.844433    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 15:39:01.879855 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.879825    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b444dd987-vqlxd"]
Sep 17 15:39:01.883099 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.883084    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:01.885178 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.885141    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 15:39:01.885178 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.885176    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 15:39:01.885417 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.885398    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fl2w8\""
Sep 17 15:39:01.886108 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.886092    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 15:39:01.891215 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.891192    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 15:39:01.898288 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.898267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b444dd987-vqlxd"]
Sep 17 15:39:01.901588 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.901537    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sfthf"]
Sep 17 15:39:01.905351 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.905331    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jdr9w"]
Sep 17 15:39:01.905496 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.905480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:01.907292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.907273    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 15:39:01.907521 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.907506    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 15:39:01.907844 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.907825    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghpsh\""
Sep 17 15:39:01.908774 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.908723    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jdr9w"
Sep 17 15:39:01.910600 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.910581    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 15:39:01.910600 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.910603    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 15:39:01.910774 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.910665    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 15:39:01.910774 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.910754    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r2vb9\""
Sep 17 15:39:01.914834 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.914816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jdr9w"]
Sep 17 15:39:01.914964 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.914942    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sfthf"]
Sep 17 15:39:01.932201 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.932180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:01.932302 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.932238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455c538-28c2-4ae8-957f-eea968f8ce7a-ca-trust-extracted\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:01.932359 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.932305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-certificates\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:01.932359 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.932331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-installation-pull-secrets\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:01.932440 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.932358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj47s\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-kube-api-access-cj47s\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:01.932440 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.932402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-bound-sa-token\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:01.932561 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.932435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-image-registry-private-configuration\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:01.932561 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:01.932523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-trusted-ca\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.033172 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-certificates\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.033172 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033171    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-installation-pull-secrets\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.033398 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj47s\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-kube-api-access-cj47s\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.033398 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25033f9-7df1-47bb-a41a-3e3a57d36da5-cert\") pod \"ingress-canary-jdr9w\" (UID: \"d25033f9-7df1-47bb-a41a-3e3a57d36da5\") " pod="openshift-ingress-canary/ingress-canary-jdr9w"
Sep 17 15:39:02.033398 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033247    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-bound-sa-token\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.033398 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc7d157-3364-4dde-93a0-06e97da1db07-config-volume\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.033398 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033304    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-image-registry-private-configuration\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.033398 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-trusted-ca\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.033398 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033404    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc7d157-3364-4dde-93a0-06e97da1db07-tmp-dir\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx88t\" (UniqueName: \"kubernetes.io/projected/0dc7d157-3364-4dde-93a0-06e97da1db07-kube-api-access-qx88t\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455c538-28c2-4ae8-957f-eea968f8ce7a-ca-trust-extracted\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033568    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxg29\" (UniqueName: \"kubernetes.io/projected/d25033f9-7df1-47bb-a41a-3e3a57d36da5-kube-api-access-lxg29\") pod \"ingress-canary-jdr9w\" (UID: \"d25033f9-7df1-47bb-a41a-3e3a57d36da5\") " pod="openshift-ingress-canary/ingress-canary-jdr9w"
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.033643    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.033661    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b444dd987-vqlxd: secret "image-registry-tls" not found
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.033733    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls podName:6455c538-28c2-4ae8-957f-eea968f8ce7a nodeName:}" failed. No retries permitted until 2026-09-17 15:39:02.533705223 +0000 UTC m=+33.457279782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls") pod "image-registry-7b444dd987-vqlxd" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a") : secret "image-registry-tls" not found
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033959    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455c538-28c2-4ae8-957f-eea968f8ce7a-ca-trust-extracted\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.034079 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.033979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-certificates\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.034617 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.034587    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-trusted-ca\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.038780 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.038756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-installation-pull-secrets\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.039153 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.039130    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-image-registry-private-configuration\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.043536 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.043513    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj47s\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-kube-api-access-cj47s\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.045783 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.045761    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-bound-sa-token\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.134429 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.134346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25033f9-7df1-47bb-a41a-3e3a57d36da5-cert\") pod \"ingress-canary-jdr9w\" (UID: \"d25033f9-7df1-47bb-a41a-3e3a57d36da5\") " pod="openshift-ingress-canary/ingress-canary-jdr9w"
Sep 17 15:39:02.134429 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.134389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc7d157-3364-4dde-93a0-06e97da1db07-config-volume\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.134654 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.134437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.134654 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.134454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc7d157-3364-4dde-93a0-06e97da1db07-tmp-dir\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.134654 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.134499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx88t\" (UniqueName: \"kubernetes.io/projected/0dc7d157-3364-4dde-93a0-06e97da1db07-kube-api-access-qx88t\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.134654 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.134519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxg29\" (UniqueName: \"kubernetes.io/projected/d25033f9-7df1-47bb-a41a-3e3a57d36da5-kube-api-access-lxg29\") pod \"ingress-canary-jdr9w\" (UID: \"d25033f9-7df1-47bb-a41a-3e3a57d36da5\") " pod="openshift-ingress-canary/ingress-canary-jdr9w"
Sep 17 15:39:02.134845 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.134691    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:02.134845 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.134777    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls podName:0dc7d157-3364-4dde-93a0-06e97da1db07 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:02.634755727 +0000 UTC m=+33.558330286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls") pod "dns-default-sfthf" (UID: "0dc7d157-3364-4dde-93a0-06e97da1db07") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:02.134933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.134852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dc7d157-3364-4dde-93a0-06e97da1db07-tmp-dir\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.135130 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.135106    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0dc7d157-3364-4dde-93a0-06e97da1db07-config-volume\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.136752 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.136730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d25033f9-7df1-47bb-a41a-3e3a57d36da5-cert\") pod \"ingress-canary-jdr9w\" (UID: \"d25033f9-7df1-47bb-a41a-3e3a57d36da5\") " pod="openshift-ingress-canary/ingress-canary-jdr9w"
Sep 17 15:39:02.142458 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.142436    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxg29\" (UniqueName: \"kubernetes.io/projected/d25033f9-7df1-47bb-a41a-3e3a57d36da5-kube-api-access-lxg29\") pod \"ingress-canary-jdr9w\" (UID: \"d25033f9-7df1-47bb-a41a-3e3a57d36da5\") " pod="openshift-ingress-canary/ingress-canary-jdr9w"
Sep 17 15:39:02.142562 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.142509    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx88t\" (UniqueName: \"kubernetes.io/projected/0dc7d157-3364-4dde-93a0-06e97da1db07-kube-api-access-qx88t\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.224880 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.224845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jdr9w"
Sep 17 15:39:02.235592 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.235563    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:39:02.235717 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.235693    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:02.235770 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.235764    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs podName:4421c85c-5367-4632-b2c3-f91661885873 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:34.235745342 +0000 UTC m=+65.159319895 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs") pod "network-metrics-daemon-lhp55" (UID: "4421c85c-5367-4632-b2c3-f91661885873") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 15:39:02.336530 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.336487    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:39:02.336673 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.336644    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:02.336745 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.336733    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret podName:0f9ba41d-7cfb-423d-acbb-8fee6d495fe7 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:34.336706372 +0000 UTC m=+65.260280920 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret") pod "global-pull-secret-syncer-8lfn8" (UID: "0f9ba41d-7cfb-423d-acbb-8fee6d495fe7") : object "kube-system"/"original-pull-secret" not registered
Sep 17 15:39:02.437591 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.437519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:39:02.437735 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.437640    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 15:39:02.437735 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.437659    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 15:39:02.437735 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.437672    2581 projected.go:196] Error preparing data for projected volume kube-api-access-t9jxw for pod openshift-network-diagnostics/network-check-target-b6wrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:02.437735 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.437732    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw podName:7fcd3586-396b-4b4d-945d-cc0f3853721b nodeName:}" failed. No retries permitted until 2026-09-17 15:39:34.437714527 +0000 UTC m=+65.361289084 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t9jxw" (UniqueName: "kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw") pod "network-check-target-b6wrg" (UID: "7fcd3586-396b-4b4d-945d-cc0f3853721b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 15:39:02.538454 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.538409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:02.538631 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.538585    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:02.538631 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.538605    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b444dd987-vqlxd: secret "image-registry-tls" not found
Sep 17 15:39:02.538740 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.538680    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls podName:6455c538-28c2-4ae8-957f-eea968f8ce7a nodeName:}" failed. No retries permitted until 2026-09-17 15:39:03.538660231 +0000 UTC m=+34.462234800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls") pod "image-registry-7b444dd987-vqlxd" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a") : secret "image-registry-tls" not found
Sep 17 15:39:02.552477 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.552437    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:39:02.552623 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.552490    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:39:02.552623 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.552555    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:39:02.554704 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.554679    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:39:02.554812 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.554711    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h48gq\""
Sep 17 15:39:02.554918 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.554899    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:39:02.555001 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.554950    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:39:02.555001 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.554988    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:39:02.555103 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.555042    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k8wzk\""
Sep 17 15:39:02.638852 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:02.638815    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:02.639028 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.639002    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:02.639108 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:02.639078    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls podName:0dc7d157-3364-4dde-93a0-06e97da1db07 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:03.639058887 +0000 UTC m=+34.562633449 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls") pod "dns-default-sfthf" (UID: "0dc7d157-3364-4dde-93a0-06e97da1db07") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:03.546583 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:03.546545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:03.546983 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:03.546735    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:03.546983 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:03.546750    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b444dd987-vqlxd: secret "image-registry-tls" not found
Sep 17 15:39:03.546983 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:03.546806    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls podName:6455c538-28c2-4ae8-957f-eea968f8ce7a nodeName:}" failed. No retries permitted until 2026-09-17 15:39:05.546789425 +0000 UTC m=+36.470363988 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls") pod "image-registry-7b444dd987-vqlxd" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a") : secret "image-registry-tls" not found
Sep 17 15:39:03.647887 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:03.647849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:03.648044 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:03.647997    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:03.648086 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:03.648058    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls podName:0dc7d157-3364-4dde-93a0-06e97da1db07 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:05.648043933 +0000 UTC m=+36.571618478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls") pod "dns-default-sfthf" (UID: "0dc7d157-3364-4dde-93a0-06e97da1db07") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:03.960290 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:03.960216    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jdr9w"]
Sep 17 15:39:03.974639 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:39:03.974606    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25033f9_7df1_47bb_a41a_3e3a57d36da5.slice/crio-1c4c055ffd2080b53ecdd388570df59d6d0c08328f31aae99e73ae85f4d12c90 WatchSource:0}: Error finding container 1c4c055ffd2080b53ecdd388570df59d6d0c08328f31aae99e73ae85f4d12c90: Status 404 returned error can't find the container with id 1c4c055ffd2080b53ecdd388570df59d6d0c08328f31aae99e73ae85f4d12c90
Sep 17 15:39:04.686891 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:04.686861    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9f08147-9fea-4b2f-a2d7-59d0796228b2" containerID="8c44ce37e373b3de353188d1fd441c2472020a441c1609b268cfc7a711de8e00" exitCode=0
Sep 17 15:39:04.687289 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:04.686956    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcjdl" event={"ID":"e9f08147-9fea-4b2f-a2d7-59d0796228b2","Type":"ContainerDied","Data":"8c44ce37e373b3de353188d1fd441c2472020a441c1609b268cfc7a711de8e00"}
Sep 17 15:39:04.688119 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:04.688100    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jdr9w" event={"ID":"d25033f9-7df1-47bb-a41a-3e3a57d36da5","Type":"ContainerStarted","Data":"1c4c055ffd2080b53ecdd388570df59d6d0c08328f31aae99e73ae85f4d12c90"}
Sep 17 15:39:05.563957 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:05.563925    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:05.564111 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:05.564059    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:05.564111 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:05.564073    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b444dd987-vqlxd: secret "image-registry-tls" not found
Sep 17 15:39:05.564181 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:05.564122    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls podName:6455c538-28c2-4ae8-957f-eea968f8ce7a nodeName:}" failed. No retries permitted until 2026-09-17 15:39:09.564106341 +0000 UTC m=+40.487680889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls") pod "image-registry-7b444dd987-vqlxd" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a") : secret "image-registry-tls" not found
Sep 17 15:39:05.664473 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:05.664427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:05.664637 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:05.664563    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:05.664637 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:05.664622    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls podName:0dc7d157-3364-4dde-93a0-06e97da1db07 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:09.664608029 +0000 UTC m=+40.588182573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls") pod "dns-default-sfthf" (UID: "0dc7d157-3364-4dde-93a0-06e97da1db07") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:05.694439 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:05.694409    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9f08147-9fea-4b2f-a2d7-59d0796228b2" containerID="c81216526fe6af112aa612173383a976342f08ea4f7bdb89d147d7ce3c6efc7f" exitCode=0
Sep 17 15:39:05.694749 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:05.694454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcjdl" event={"ID":"e9f08147-9fea-4b2f-a2d7-59d0796228b2","Type":"ContainerDied","Data":"c81216526fe6af112aa612173383a976342f08ea4f7bdb89d147d7ce3c6efc7f"}
Sep 17 15:39:06.699995 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:06.699958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dcjdl" event={"ID":"e9f08147-9fea-4b2f-a2d7-59d0796228b2","Type":"ContainerStarted","Data":"029c381b5b969d4f3bf11f1a2b62ce8c752bbb12adf2d865e3929d85b322509d"}
Sep 17 15:39:06.721181 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:06.721132    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dcjdl" podStartSLOduration=6.689332352 podStartE2EDuration="37.721118219s" totalImagesPullingTime="31.031785867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:06.720477565 +0000 UTC m=+37.644052123" watchObservedRunningTime="2026-09-17 15:39:06.721118219 +0000 UTC m=+37.644692777"
Sep 17 15:39:07.703600 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:07.703557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jdr9w" event={"ID":"d25033f9-7df1-47bb-a41a-3e3a57d36da5","Type":"ContainerStarted","Data":"d39cd43bee53d029eb24a3f60efca0cf873a29c37bab96195324c283572faabe"}
Sep 17 15:39:07.719948 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:07.719899    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jdr9w" podStartSLOduration=3.616383841 podStartE2EDuration="6.719883322s" totalImagesPullingTime="3.103499481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:07.718957942 +0000 UTC m=+38.642532509" watchObservedRunningTime="2026-09-17 15:39:07.719883322 +0000 UTC m=+38.643457888"
Sep 17 15:39:09.592367 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:09.592330    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:09.592779 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:09.592495    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:09.592779 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:09.592511    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b444dd987-vqlxd: secret "image-registry-tls" not found
Sep 17 15:39:09.592779 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:09.592566    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls podName:6455c538-28c2-4ae8-957f-eea968f8ce7a nodeName:}" failed. No retries permitted until 2026-09-17 15:39:17.59254956 +0000 UTC m=+48.516124105 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls") pod "image-registry-7b444dd987-vqlxd" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a") : secret "image-registry-tls" not found
Sep 17 15:39:09.696308 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:09.694081    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:09.696308 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:09.694268    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:09.696308 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:09.694356    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls podName:0dc7d157-3364-4dde-93a0-06e97da1db07 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:17.694336186 +0000 UTC m=+48.617910748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls") pod "dns-default-sfthf" (UID: "0dc7d157-3364-4dde-93a0-06e97da1db07") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:17.655164 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:17.655126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:17.655601 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:17.655268    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:17.655601 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:17.655285    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b444dd987-vqlxd: secret "image-registry-tls" not found
Sep 17 15:39:17.655601 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:17.655355    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls podName:6455c538-28c2-4ae8-957f-eea968f8ce7a nodeName:}" failed. No retries permitted until 2026-09-17 15:39:33.655337402 +0000 UTC m=+64.578911967 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls") pod "image-registry-7b444dd987-vqlxd" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a") : secret "image-registry-tls" not found
Sep 17 15:39:17.755640 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:17.755613    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:17.755768 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:17.755752    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:17.755832 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:17.755822    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls podName:0dc7d157-3364-4dde-93a0-06e97da1db07 nodeName:}" failed. No retries permitted until 2026-09-17 15:39:33.755803805 +0000 UTC m=+64.679378355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls") pod "dns-default-sfthf" (UID: "0dc7d157-3364-4dde-93a0-06e97da1db07") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:28.679970 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:28.679939    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-486bt"
Sep 17 15:39:33.674444 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:33.674402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:39:33.674829 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:33.674556    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:39:33.674829 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:33.674570    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b444dd987-vqlxd: secret "image-registry-tls" not found
Sep 17 15:39:33.674829 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:33.674627    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls podName:6455c538-28c2-4ae8-957f-eea968f8ce7a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:05.674608946 +0000 UTC m=+96.598183498 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls") pod "image-registry-7b444dd987-vqlxd" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a") : secret "image-registry-tls" not found
Sep 17 15:39:33.775379 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:33.775351    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:39:33.775546 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:33.775475    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:39:33.775546 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:33.775532    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls podName:0dc7d157-3364-4dde-93a0-06e97da1db07 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:05.775519995 +0000 UTC m=+96.699094539 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls") pod "dns-default-sfthf" (UID: "0dc7d157-3364-4dde-93a0-06e97da1db07") : secret "dns-default-metrics-tls" not found
Sep 17 15:39:34.278820 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.278783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:39:34.280877 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.280859    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 15:39:34.289518 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:34.289501    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 15:39:34.289606 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:39:34.289570    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs podName:4421c85c-5367-4632-b2c3-f91661885873 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:38.289548357 +0000 UTC m=+129.213122915 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs") pod "network-metrics-daemon-lhp55" (UID: "4421c85c-5367-4632-b2c3-f91661885873") : secret "metrics-daemon-secret" not found
Sep 17 15:39:34.379364 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.379326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:39:34.381180 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.381161    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 15:39:34.391655 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.391628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0f9ba41d-7cfb-423d-acbb-8fee6d495fe7-original-pull-secret\") pod \"global-pull-secret-syncer-8lfn8\" (UID: \"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7\") " pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:39:34.479897 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.479871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:39:34.481924 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.481907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 15:39:34.492038 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.492014    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 15:39:34.503375 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.503347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9jxw\" (UniqueName: \"kubernetes.io/projected/7fcd3586-396b-4b4d-945d-cc0f3853721b-kube-api-access-t9jxw\") pod \"network-check-target-b6wrg\" (UID: \"7fcd3586-396b-4b4d-945d-cc0f3853721b\") " pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:39:34.666509 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.666453    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-k8wzk\""
Sep 17 15:39:34.671964 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.671949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8lfn8"
Sep 17 15:39:34.675079 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.675064    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:39:34.953850 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.953782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b6wrg"]
Sep 17 15:39:34.956477 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:34.956435    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8lfn8"]
Sep 17 15:39:34.957094 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:39:34.957073    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcd3586_396b_4b4d_945d_cc0f3853721b.slice/crio-ec18469863b17b8886678002a9dd14836be173937c3c437dc969fe2577f6d0b1 WatchSource:0}: Error finding container ec18469863b17b8886678002a9dd14836be173937c3c437dc969fe2577f6d0b1: Status 404 returned error can't find the container with id ec18469863b17b8886678002a9dd14836be173937c3c437dc969fe2577f6d0b1
Sep 17 15:39:34.959680 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:39:34.959661    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9ba41d_7cfb_423d_acbb_8fee6d495fe7.slice/crio-49e290e6ebc8f53e356b8176f49caf696dd7b5d9e68a55aecb835a9b30250ce4 WatchSource:0}: Error finding container 49e290e6ebc8f53e356b8176f49caf696dd7b5d9e68a55aecb835a9b30250ce4: Status 404 returned error can't find the container with id 49e290e6ebc8f53e356b8176f49caf696dd7b5d9e68a55aecb835a9b30250ce4
Sep 17 15:39:35.770149 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:35.770108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b6wrg" event={"ID":"7fcd3586-396b-4b4d-945d-cc0f3853721b","Type":"ContainerStarted","Data":"ec18469863b17b8886678002a9dd14836be173937c3c437dc969fe2577f6d0b1"}
Sep 17 15:39:35.772007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:35.771953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8lfn8" event={"ID":"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7","Type":"ContainerStarted","Data":"49e290e6ebc8f53e356b8176f49caf696dd7b5d9e68a55aecb835a9b30250ce4"}
Sep 17 15:39:40.786671 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:40.786630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b6wrg" event={"ID":"7fcd3586-396b-4b4d-945d-cc0f3853721b","Type":"ContainerStarted","Data":"2a973b37ca3d10b590b980870ff8a0ba5286889416be096aba373d608930c1a2"}
Sep 17 15:39:40.787133 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:40.786714    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:39:40.788095 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:40.788074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8lfn8" event={"ID":"0f9ba41d-7cfb-423d-acbb-8fee6d495fe7","Type":"ContainerStarted","Data":"401cd92d8a553b8554c8c3a78699552091a343df7c327113f8bd67429f1a805a"}
Sep 17 15:39:40.801522 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:40.801473    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b6wrg" podStartSLOduration=65.653857434 podStartE2EDuration="1m10.801447195s" totalImagesPullingTime="5.147589761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:40.801383413 +0000 UTC m=+71.724957980" watchObservedRunningTime="2026-09-17 15:39:40.801447195 +0000 UTC m=+71.725021764"
Sep 17 15:39:40.815138 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:39:40.815097    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8lfn8" podStartSLOduration=65.665205444 podStartE2EDuration="1m10.81508828s" totalImagesPullingTime="5.149882836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:39:40.814413829 +0000 UTC m=+71.737988396" watchObservedRunningTime="2026-09-17 15:39:40.81508828 +0000 UTC m=+71.738662846"
Sep 17 15:40:05.696147 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:05.696104    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") pod \"image-registry-7b444dd987-vqlxd\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") " pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:40:05.696667 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:05.696259    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 15:40:05.696667 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:05.696278    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b444dd987-vqlxd: secret "image-registry-tls" not found
Sep 17 15:40:05.696667 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:05.696346    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls podName:6455c538-28c2-4ae8-957f-eea968f8ce7a nodeName:}" failed. No retries permitted until 2026-09-17 15:41:09.696328191 +0000 UTC m=+160.619902740 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls") pod "image-registry-7b444dd987-vqlxd" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a") : secret "image-registry-tls" not found
Sep 17 15:40:05.797322 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:05.797278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:40:05.797515 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:05.797398    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 15:40:05.797515 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:05.797485    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls podName:0dc7d157-3364-4dde-93a0-06e97da1db07 nodeName:}" failed. No retries permitted until 2026-09-17 15:41:09.797448829 +0000 UTC m=+160.721023374 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls") pod "dns-default-sfthf" (UID: "0dc7d157-3364-4dde-93a0-06e97da1db07") : secret "dns-default-metrics-tls" not found
Sep 17 15:40:08.032785 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.032750    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-648db6667b-2h95h"]
Sep 17 15:40:08.035990 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.035967    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.038874 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.038854    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 15:40:08.039275 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.039255    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 15:40:08.039314 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.039271    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 15:40:08.039314 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.039289    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 15:40:08.039590 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.039561    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 15:40:08.039999 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.039982    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 15:40:08.040274 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.040260    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cpjhb\""
Sep 17 15:40:08.054193 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.054171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-648db6667b-2h95h"]
Sep 17 15:40:08.113909 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.113878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.114072 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.113913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-default-certificate\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.114072 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.113931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k6lj\" (UniqueName: \"kubernetes.io/projected/b4db3ce1-346c-42de-8ca6-240881068330-kube-api-access-6k6lj\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.114072 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.113964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-stats-auth\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.114072 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.114048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.214886 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.214844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.214886 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.214889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-default-certificate\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.215113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.214907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k6lj\" (UniqueName: \"kubernetes.io/projected/b4db3ce1-346c-42de-8ca6-240881068330-kube-api-access-6k6lj\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.215113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.214941    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-stats-auth\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.215113 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:08.215060    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:08.715038091 +0000 UTC m=+99.638612654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:08.215259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.215117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.215299 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:08.215279    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:08.215375 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:08.215355    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:08.715338472 +0000 UTC m=+99.638913027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : secret "router-metrics-certs-default" not found
Sep 17 15:40:08.217430 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.217399    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-default-certificate\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.217874 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.217858    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-stats-auth\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.222798 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.222776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k6lj\" (UniqueName: \"kubernetes.io/projected/b4db3ce1-346c-42de-8ca6-240881068330-kube-api-access-6k6lj\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.718948 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.718910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.719123 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:08.718980    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:08.719123 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:08.719102    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:09.719085494 +0000 UTC m=+100.642660039 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:08.719201 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:08.719125    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:08.719201 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:08.719182    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:09.71916927 +0000 UTC m=+100.642743815 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : secret "router-metrics-certs-default" not found
Sep 17 15:40:09.726099 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:09.726071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:09.726515 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:09.726126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:09.726515 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:09.726242    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:11.726220005 +0000 UTC m=+102.649794557 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:09.726515 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:09.726264    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:09.726515 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:09.726308    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:11.726293521 +0000 UTC m=+102.649868070 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : secret "router-metrics-certs-default" not found
Sep 17 15:40:11.742847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:11.742814    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:11.743200 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:11.742870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:11.743200 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:11.742955    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:11.743200 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:11.742980    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:15.742967355 +0000 UTC m=+106.666541913 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:11.743200 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:11.743001    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:15.742989349 +0000 UTC m=+106.666563899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : secret "router-metrics-certs-default" not found
Sep 17 15:40:11.792941 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:11.792918    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b6wrg"
Sep 17 15:40:12.543299 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.543267    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wn4ng"]
Sep 17 15:40:12.577203 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.577176    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wn4ng"]
Sep 17 15:40:12.577341 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.577289    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.579049 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.579025    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 15:40:12.579183 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.579158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 15:40:12.579429 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.579412    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 15:40:12.579544 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.579526    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 15:40:12.579698 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.579686    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9r9ks\""
Sep 17 15:40:12.652789 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.652767    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f4856dd-06b6-4154-bcdb-569255b6c63a-crio-socket\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.652923 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.652795    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jz7t\" (UniqueName: \"kubernetes.io/projected/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-api-access-9jz7t\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.652923 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.652834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4856dd-06b6-4154-bcdb-569255b6c63a-data-volume\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.652923 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.652878    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.652923 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.652908    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.753728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.753701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4856dd-06b6-4154-bcdb-569255b6c63a-data-volume\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.753743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.753762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.753822    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f4856dd-06b6-4154-bcdb-569255b6c63a-crio-socket\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.753838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jz7t\" (UniqueName: \"kubernetes.io/projected/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-api-access-9jz7t\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:12.753846    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:12.753923    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.25390149 +0000 UTC m=+104.177476034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.753927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3f4856dd-06b6-4154-bcdb-569255b6c63a-crio-socket\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:12.753945    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:12.754014    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:13.253997155 +0000 UTC m=+104.177571717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:12.754121 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.754024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f4856dd-06b6-4154-bcdb-569255b6c63a-data-volume\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:12.766780 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:12.766756    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jz7t\" (UniqueName: \"kubernetes.io/projected/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-api-access-9jz7t\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:13.257569 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:13.257536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:13.257569 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:13.257569    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:13.257782 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:13.257656    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:13.257782 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:13.257686    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:13.257782 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:13.257715    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:14.257700085 +0000 UTC m=+105.181274635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:13.257782 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:13.257730    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:14.257724134 +0000 UTC m=+105.181298678 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:13.806739 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:13.806705    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk"]
Sep 17 15:40:13.833517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:13.833488    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk"]
Sep 17 15:40:13.833668 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:13.833576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk"
Sep 17 15:40:13.837034 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:13.837014    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 15:40:13.837413 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:13.837395    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 15:40:13.837484 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:13.837427    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fqzg7\""
Sep 17 15:40:13.964172 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:13.964145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzc8k\" (UniqueName: \"kubernetes.io/projected/f6874331-4a57-439f-9665-0a269c3f40f4-kube-api-access-kzc8k\") pod \"migrator-8b4d697c5-2ksvk\" (UID: \"f6874331-4a57-439f-9665-0a269c3f40f4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk"
Sep 17 15:40:14.064707 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:14.064639    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzc8k\" (UniqueName: \"kubernetes.io/projected/f6874331-4a57-439f-9665-0a269c3f40f4-kube-api-access-kzc8k\") pod \"migrator-8b4d697c5-2ksvk\" (UID: \"f6874331-4a57-439f-9665-0a269c3f40f4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk"
Sep 17 15:40:14.073184 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:14.073164    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzc8k\" (UniqueName: \"kubernetes.io/projected/f6874331-4a57-439f-9665-0a269c3f40f4-kube-api-access-kzc8k\") pod \"migrator-8b4d697c5-2ksvk\" (UID: \"f6874331-4a57-439f-9665-0a269c3f40f4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk"
Sep 17 15:40:14.142539 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:14.142513    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk"
Sep 17 15:40:14.266248 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:14.266217    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:14.266248 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:14.266250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:14.266420 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:14.266344    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:14.266420 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:14.266390    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:14.266420 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:14.266409    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:16.266392697 +0000 UTC m=+107.189967243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:14.266571 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:14.266479    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:16.26645045 +0000 UTC m=+107.190025009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:14.318253 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:14.318199    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7gbxk_d033245e-d93b-4b38-b1c8-62351fd7cfc1/dns-node-resolver/0.log"
Sep 17 15:40:14.420075 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:14.420050    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk"]
Sep 17 15:40:14.423223 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:14.423193    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6874331_4a57_439f_9665_0a269c3f40f4.slice/crio-ef505d7dd7b586b54de68d7d3db8ae179a605a64fa06497ec3be720a9ae9dff0 WatchSource:0}: Error finding container ef505d7dd7b586b54de68d7d3db8ae179a605a64fa06497ec3be720a9ae9dff0: Status 404 returned error can't find the container with id ef505d7dd7b586b54de68d7d3db8ae179a605a64fa06497ec3be720a9ae9dff0
Sep 17 15:40:14.867936 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:14.867896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk" event={"ID":"f6874331-4a57-439f-9665-0a269c3f40f4","Type":"ContainerStarted","Data":"ef505d7dd7b586b54de68d7d3db8ae179a605a64fa06497ec3be720a9ae9dff0"}
Sep 17 15:40:15.319073 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:15.319032    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-48hkv_6b5af873-2e95-452f-9eb6-936d70e56b0a/node-ca/0.log"
Sep 17 15:40:15.776827 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:15.776751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:15.776827 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:15.776811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:15.777003 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:15.776895    2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 15:40:15.777003 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:15.776963    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:23.776931722 +0000 UTC m=+114.700506268 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : configmap references non-existent config key: service-ca.crt
Sep 17 15:40:15.777003 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:15.776984    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs podName:b4db3ce1-346c-42de-8ca6-240881068330 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:23.776977822 +0000 UTC m=+114.700552370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs") pod "router-default-648db6667b-2h95h" (UID: "b4db3ce1-346c-42de-8ca6-240881068330") : secret "router-metrics-certs-default" not found
Sep 17 15:40:15.875389 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:15.875355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk" event={"ID":"f6874331-4a57-439f-9665-0a269c3f40f4","Type":"ContainerStarted","Data":"6fa6a1f57cebbc17d697b454cb47213b01e4c1a93fbb48d419f6da2b6d97ef24"}
Sep 17 15:40:15.875839 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:15.875446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk" event={"ID":"f6874331-4a57-439f-9665-0a269c3f40f4","Type":"ContainerStarted","Data":"53c68c5c25557a98a8b361e0d9eb8bbb55cfe1d97b09d57b1a25e4a32e4e5859"}
Sep 17 15:40:15.891342 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:15.891294    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-2ksvk" podStartSLOduration=1.874276469 podStartE2EDuration="2.891281097s" totalImagesPullingTime="1.017004628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:15.890047585 +0000 UTC m=+106.813622152" watchObservedRunningTime="2026-09-17 15:40:15.891281097 +0000 UTC m=+106.814855663"
Sep 17 15:40:16.280874 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:16.280833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:16.280874 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:16.280883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:16.281186 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:16.280953    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:16.281186 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:16.280992    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:16.281186 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:16.281024    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:20.281010458 +0000 UTC m=+111.204585003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:16.281186 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:16.281053    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:20.281037257 +0000 UTC m=+111.204611819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:16.317965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:16.317943    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jdr9w_d25033f9-7df1-47bb-a41a-3e3a57d36da5/serve-healthcheck-canary/0.log"
Sep 17 15:40:20.310297 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:20.310257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:20.310297 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:20.310298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:20.310749 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:20.310384    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 15:40:20.310749 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:20.310423    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:20.310749 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:20.310451    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:28.310435713 +0000 UTC m=+119.234010281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : configmap "kube-rbac-proxy" not found
Sep 17 15:40:20.310749 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:20.310493    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls podName:3f4856dd-06b6-4154-bcdb-569255b6c63a nodeName:}" failed. No retries permitted until 2026-09-17 15:40:28.310482654 +0000 UTC m=+119.234057199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wn4ng" (UID: "3f4856dd-06b6-4154-bcdb-569255b6c63a") : secret "insights-runtime-extractor-tls" not found
Sep 17 15:40:23.840094 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:23.840045    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:23.840503 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:23.840226    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:23.840965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:23.840947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4db3ce1-346c-42de-8ca6-240881068330-service-ca-bundle\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:23.842439 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:23.842419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b4db3ce1-346c-42de-8ca6-240881068330-metrics-certs\") pod \"router-default-648db6667b-2h95h\" (UID: \"b4db3ce1-346c-42de-8ca6-240881068330\") " pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:23.945451 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:23.945409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:24.213921 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:24.213854    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-648db6667b-2h95h"]
Sep 17 15:40:24.217699 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:24.217673    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4db3ce1_346c_42de_8ca6_240881068330.slice/crio-0fa9b8a836036f4b99f173a487fd04575b6d73348e3739a69baedf8fb871b365 WatchSource:0}: Error finding container 0fa9b8a836036f4b99f173a487fd04575b6d73348e3739a69baedf8fb871b365: Status 404 returned error can't find the container with id 0fa9b8a836036f4b99f173a487fd04575b6d73348e3739a69baedf8fb871b365
Sep 17 15:40:24.899978 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:24.899942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-648db6667b-2h95h" event={"ID":"b4db3ce1-346c-42de-8ca6-240881068330","Type":"ContainerStarted","Data":"618d51103d306675eb3414f52f6cc00333745d5210efc91d0d1dc0efafa87104"}
Sep 17 15:40:24.900368 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:24.899988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-648db6667b-2h95h" event={"ID":"b4db3ce1-346c-42de-8ca6-240881068330","Type":"ContainerStarted","Data":"0fa9b8a836036f4b99f173a487fd04575b6d73348e3739a69baedf8fb871b365"}
Sep 17 15:40:24.920239 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:24.920189    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-648db6667b-2h95h" podStartSLOduration=16.920173826 podStartE2EDuration="16.920173826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:24.919387747 +0000 UTC m=+115.842962315" watchObservedRunningTime="2026-09-17 15:40:24.920173826 +0000 UTC m=+115.843748393"
Sep 17 15:40:24.945696 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:24.945669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:24.948153 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:24.948130    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:25.903032 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:25.902997    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:25.904171 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:25.904149    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-648db6667b-2h95h"
Sep 17 15:40:28.377551 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:28.377516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:28.377551 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:28.377553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:28.378141 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:28.378118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3f4856dd-06b6-4154-bcdb-569255b6c63a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:28.379712 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:28.379683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3f4856dd-06b6-4154-bcdb-569255b6c63a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wn4ng\" (UID: \"3f4856dd-06b6-4154-bcdb-569255b6c63a\") " pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:28.486217 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:28.486186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wn4ng"
Sep 17 15:40:28.753390 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:28.753316    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wn4ng"]
Sep 17 15:40:28.756395 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:28.756372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4856dd_06b6_4154_bcdb_569255b6c63a.slice/crio-744f347e41b73f8790202fe3ca632f03fb12a5afb6323bad834ab22b5713cb5a WatchSource:0}: Error finding container 744f347e41b73f8790202fe3ca632f03fb12a5afb6323bad834ab22b5713cb5a: Status 404 returned error can't find the container with id 744f347e41b73f8790202fe3ca632f03fb12a5afb6323bad834ab22b5713cb5a
Sep 17 15:40:28.913783 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:28.913753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wn4ng" event={"ID":"3f4856dd-06b6-4154-bcdb-569255b6c63a","Type":"ContainerStarted","Data":"dbdbab76a075cba668297554e319f9bb221a3311f07e9f624f7e9f27467385c4"}
Sep 17 15:40:28.913937 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:28.913824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wn4ng" event={"ID":"3f4856dd-06b6-4154-bcdb-569255b6c63a","Type":"ContainerStarted","Data":"744f347e41b73f8790202fe3ca632f03fb12a5afb6323bad834ab22b5713cb5a"}
Sep 17 15:40:29.917855 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:29.917821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wn4ng" event={"ID":"3f4856dd-06b6-4154-bcdb-569255b6c63a","Type":"ContainerStarted","Data":"af02464d24c9835a3449ad21c360d02c8ef9b44c7a7885cf54766545304290e4"}
Sep 17 15:40:31.926521 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:31.926484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wn4ng" event={"ID":"3f4856dd-06b6-4154-bcdb-569255b6c63a","Type":"ContainerStarted","Data":"2efb6f71141f68813ddb611e9d716f84e0509de31a6fcaa962d1bc3b7abdb927"}
Sep 17 15:40:31.944392 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:31.944351    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wn4ng" podStartSLOduration=17.659160393 podStartE2EDuration="19.944337969s" totalImagesPullingTime="2.285177576s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:31.943623684 +0000 UTC m=+122.867198261" watchObservedRunningTime="2026-09-17 15:40:31.944337969 +0000 UTC m=+122.867912536"
Sep 17 15:40:34.180146 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.180113    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"]
Sep 17 15:40:34.183192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.183173    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"
Sep 17 15:40:34.185730 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.185712    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-r2xf6\""
Sep 17 15:40:34.185730 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.185719    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 15:40:34.194032 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.194013    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"]
Sep 17 15:40:34.321795 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.321757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ee68ea93-3b67-41d4-a7af-5505da06106d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-d5587\" (UID: \"ee68ea93-3b67-41d4-a7af-5505da06106d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"
Sep 17 15:40:34.422574 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.422482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ee68ea93-3b67-41d4-a7af-5505da06106d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-d5587\" (UID: \"ee68ea93-3b67-41d4-a7af-5505da06106d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"
Sep 17 15:40:34.424893 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.424871    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ee68ea93-3b67-41d4-a7af-5505da06106d-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-d5587\" (UID: \"ee68ea93-3b67-41d4-a7af-5505da06106d\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"
Sep 17 15:40:34.491994 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.491964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"
Sep 17 15:40:34.764741 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.764665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"]
Sep 17 15:40:34.767807 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:34.767780    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee68ea93_3b67_41d4_a7af_5505da06106d.slice/crio-2e9696ddb074604d43b8eea5962d9d9fa7a3241d843c7e4cf24051db1e2a44c7 WatchSource:0}: Error finding container 2e9696ddb074604d43b8eea5962d9d9fa7a3241d843c7e4cf24051db1e2a44c7: Status 404 returned error can't find the container with id 2e9696ddb074604d43b8eea5962d9d9fa7a3241d843c7e4cf24051db1e2a44c7
Sep 17 15:40:34.938044 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:34.938003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587" event={"ID":"ee68ea93-3b67-41d4-a7af-5505da06106d","Type":"ContainerStarted","Data":"2e9696ddb074604d43b8eea5962d9d9fa7a3241d843c7e4cf24051db1e2a44c7"}
Sep 17 15:40:35.942537 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:35.942435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587" event={"ID":"ee68ea93-3b67-41d4-a7af-5505da06106d","Type":"ContainerStarted","Data":"3f8f6b69e8a6c6ba70fb8e39745bb3595ae33120d684df8e7fc58158cdde7340"}
Sep 17 15:40:35.942983 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:35.942652    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"
Sep 17 15:40:35.947419 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:35.947396    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587"
Sep 17 15:40:35.956518 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:35.956453    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-d5587" podStartSLOduration=1.072708594 podStartE2EDuration="1.956440741s" totalImagesPullingTime="883.732147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:35.956262656 +0000 UTC m=+126.879837225" watchObservedRunningTime="2026-09-17 15:40:35.956440741 +0000 UTC m=+126.880015307"
Sep 17 15:40:36.378476 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.378428    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"]
Sep 17 15:40:36.381919 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.381902    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.383876 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.383851    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 15:40:36.384385 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.384360    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 17 15:40:36.384385 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.384380    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fxt6n\""
Sep 17 15:40:36.384567 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.384408    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 15:40:36.384567 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.384448    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 17 15:40:36.384567 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.384408    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 15:40:36.393060 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.393042    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"]
Sep 17 15:40:36.541634 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.541594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/694e4ba2-2ded-49b7-9084-0a86210086c4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.541869 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.541648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/694e4ba2-2ded-49b7-9084-0a86210086c4-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.541869 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.541682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/694e4ba2-2ded-49b7-9084-0a86210086c4-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.541869 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.541747    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mq4d\" (UniqueName: \"kubernetes.io/projected/694e4ba2-2ded-49b7-9084-0a86210086c4-kube-api-access-4mq4d\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.642880 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.642810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/694e4ba2-2ded-49b7-9084-0a86210086c4-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.642880 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.642850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/694e4ba2-2ded-49b7-9084-0a86210086c4-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.643042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.642889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mq4d\" (UniqueName: \"kubernetes.io/projected/694e4ba2-2ded-49b7-9084-0a86210086c4-kube-api-access-4mq4d\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.643042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.642952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/694e4ba2-2ded-49b7-9084-0a86210086c4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.643615 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.643590    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/694e4ba2-2ded-49b7-9084-0a86210086c4-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.645351 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.645322    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/694e4ba2-2ded-49b7-9084-0a86210086c4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.645434 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.645330    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/694e4ba2-2ded-49b7-9084-0a86210086c4-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.650705 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.650678    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mq4d\" (UniqueName: \"kubernetes.io/projected/694e4ba2-2ded-49b7-9084-0a86210086c4-kube-api-access-4mq4d\") pod \"prometheus-operator-68bcd7f987-rc7gk\" (UID: \"694e4ba2-2ded-49b7-9084-0a86210086c4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.690761 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.690730    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"
Sep 17 15:40:36.959126 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:36.959048    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk"]
Sep 17 15:40:36.961916 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:36.961888    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694e4ba2_2ded_49b7_9084_0a86210086c4.slice/crio-a54018b48c1b23e18ac9d78a06fc1b35eeac124756ad5300661709dd22d8ddf2 WatchSource:0}: Error finding container a54018b48c1b23e18ac9d78a06fc1b35eeac124756ad5300661709dd22d8ddf2: Status 404 returned error can't find the container with id a54018b48c1b23e18ac9d78a06fc1b35eeac124756ad5300661709dd22d8ddf2
Sep 17 15:40:37.953137 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:37.953099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk" event={"ID":"694e4ba2-2ded-49b7-9084-0a86210086c4","Type":"ContainerStarted","Data":"a54018b48c1b23e18ac9d78a06fc1b35eeac124756ad5300661709dd22d8ddf2"}
Sep 17 15:40:38.358077 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:38.358043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:40:38.360410 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:38.360381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4421c85c-5367-4632-b2c3-f91661885873-metrics-certs\") pod \"network-metrics-daemon-lhp55\" (UID: \"4421c85c-5367-4632-b2c3-f91661885873\") " pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:40:38.579448 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:38.579422    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-h48gq\""
Sep 17 15:40:38.587975 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:38.587951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lhp55"
Sep 17 15:40:38.856259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:38.856185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lhp55"]
Sep 17 15:40:38.859198 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:38.859172    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4421c85c_5367_4632_b2c3_f91661885873.slice/crio-6ca17510ee17f76bffc0e120a508d69f63572ab074555d178a2b72b01815c065 WatchSource:0}: Error finding container 6ca17510ee17f76bffc0e120a508d69f63572ab074555d178a2b72b01815c065: Status 404 returned error can't find the container with id 6ca17510ee17f76bffc0e120a508d69f63572ab074555d178a2b72b01815c065
Sep 17 15:40:38.959543 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:38.959506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhp55" event={"ID":"4421c85c-5367-4632-b2c3-f91661885873","Type":"ContainerStarted","Data":"6ca17510ee17f76bffc0e120a508d69f63572ab074555d178a2b72b01815c065"}
Sep 17 15:40:38.961244 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:38.961212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk" event={"ID":"694e4ba2-2ded-49b7-9084-0a86210086c4","Type":"ContainerStarted","Data":"4523e86382d48369474072f1f296b4a36f0c491bb0ec80eaec2a3be2e14e881f"}
Sep 17 15:40:38.961341 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:38.961254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk" event={"ID":"694e4ba2-2ded-49b7-9084-0a86210086c4","Type":"ContainerStarted","Data":"675b06b36001a5f7c29ac20baa94f1d2cb511881a87a65a087368c5959e57f2a"}
Sep 17 15:40:38.983395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:38.983342    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-rc7gk" podStartSLOduration=1.75539161 podStartE2EDuration="2.983327588s" totalImagesPullingTime="1.227935978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:38.982546385 +0000 UTC m=+129.906120949" watchObservedRunningTime="2026-09-17 15:40:38.983327588 +0000 UTC m=+129.906902154"
Sep 17 15:40:40.764992 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.764956    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"]
Sep 17 15:40:40.768156 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.768134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.770593 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.770568    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 15:40:40.770717 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.770568    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zx6bt\""
Sep 17 15:40:40.770937 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.770919    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 15:40:40.781282 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.781257    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"]
Sep 17 15:40:40.782180 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.782159    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s8bwq"]
Sep 17 15:40:40.785303 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.785286    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.787657 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.787635    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 15:40:40.787772 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.787699    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 15:40:40.787839 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.787802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 15:40:40.788000 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.787983    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7zh5h\""
Sep 17 15:40:40.875537 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.875495    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e095828-1ae8-4785-be45-76814c0d11fb-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.875677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.875552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzfk\" (UniqueName: \"kubernetes.io/projected/7e095828-1ae8-4785-be45-76814c0d11fb-kube-api-access-4lzfk\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.875677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.875589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e095828-1ae8-4785-be45-76814c0d11fb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.875677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.875657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e095828-1ae8-4785-be45-76814c0d11fb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.968955 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.968917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhp55" event={"ID":"4421c85c-5367-4632-b2c3-f91661885873","Type":"ContainerStarted","Data":"8b6b1ef2576bd9a13b9af33324a5100c253f755d94eced4adacbb597b23245e5"}
Sep 17 15:40:40.969161 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.968971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lhp55" event={"ID":"4421c85c-5367-4632-b2c3-f91661885873","Type":"ContainerStarted","Data":"953b4000fa2cc6f5e85d72434900cee1d9ece2eff2711befd42b5ee0121dcce4"}
Sep 17 15:40:40.976144 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzfk\" (UniqueName: \"kubernetes.io/projected/7e095828-1ae8-4785-be45-76814c0d11fb-kube-api-access-4lzfk\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.976144 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4646ca52-be14-4602-b081-160d471768ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.976297 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4646ca52-be14-4602-b081-160d471768ec-root\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.976297 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4646ca52-be14-4602-b081-160d471768ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.976297 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4646ca52-be14-4602-b081-160d471768ec-sys\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.976428 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4646ca52-be14-4602-b081-160d471768ec-node-exporter-textfile\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.976428 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976400    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e095828-1ae8-4785-be45-76814c0d11fb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.976515 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976423    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4646ca52-be14-4602-b081-160d471768ec-node-exporter-wtmp\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.976515 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976453    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4646ca52-be14-4602-b081-160d471768ec-node-exporter-tls\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.976588 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4646ca52-be14-4602-b081-160d471768ec-metrics-client-ca\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.976588 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e095828-1ae8-4785-be45-76814c0d11fb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.976663 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zlnk\" (UniqueName: \"kubernetes.io/projected/4646ca52-be14-4602-b081-160d471768ec-kube-api-access-4zlnk\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:40.976663 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.976654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e095828-1ae8-4785-be45-76814c0d11fb-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.977215 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.977190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7e095828-1ae8-4785-be45-76814c0d11fb-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.978760 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.978741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7e095828-1ae8-4785-be45-76814c0d11fb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.978839 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.978741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7e095828-1ae8-4785-be45-76814c0d11fb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:40.984625 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.984577    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lhp55" podStartSLOduration=130.79850094 podStartE2EDuration="2m11.984558956s" totalImagesPullingTime="1.186058016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:38:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:40.984017847 +0000 UTC m=+131.907592427" watchObservedRunningTime="2026-09-17 15:40:40.984558956 +0000 UTC m=+131.908133535"
Sep 17 15:40:40.987568 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:40.987548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzfk\" (UniqueName: \"kubernetes.io/projected/7e095828-1ae8-4785-be45-76814c0d11fb-kube-api-access-4lzfk\") pod \"openshift-state-metrics-67dcf8c979-9bp6w\" (UID: \"7e095828-1ae8-4785-be45-76814c0d11fb\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:41.077221 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"
Sep 17 15:40:41.077421 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077352    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4646ca52-be14-4602-b081-160d471768ec-metrics-client-ca\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.077525 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zlnk\" (UniqueName: \"kubernetes.io/projected/4646ca52-be14-4602-b081-160d471768ec-kube-api-access-4zlnk\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.077649 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4646ca52-be14-4602-b081-160d471768ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.077709 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4646ca52-be14-4602-b081-160d471768ec-root\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.077709 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4646ca52-be14-4602-b081-160d471768ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.077804 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4646ca52-be14-4602-b081-160d471768ec-sys\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.077804 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4646ca52-be14-4602-b081-160d471768ec-sys\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.077804 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4646ca52-be14-4602-b081-160d471768ec-root\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.077911 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077834    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4646ca52-be14-4602-b081-160d471768ec-node-exporter-textfile\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.077911 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077892    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4646ca52-be14-4602-b081-160d471768ec-node-exporter-wtmp\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.078013 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4646ca52-be14-4602-b081-160d471768ec-node-exporter-tls\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.078013 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.077943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4646ca52-be14-4602-b081-160d471768ec-metrics-client-ca\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.078119 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.078098    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4646ca52-be14-4602-b081-160d471768ec-node-exporter-wtmp\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.078174 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:41.078126    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 17 15:40:41.078174 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.078138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4646ca52-be14-4602-b081-160d471768ec-node-exporter-textfile\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.078252 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:41.078180    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4646ca52-be14-4602-b081-160d471768ec-node-exporter-tls podName:4646ca52-be14-4602-b081-160d471768ec nodeName:}" failed. No retries permitted until 2026-09-17 15:40:41.578163394 +0000 UTC m=+132.501737953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4646ca52-be14-4602-b081-160d471768ec-node-exporter-tls") pod "node-exporter-s8bwq" (UID: "4646ca52-be14-4602-b081-160d471768ec") : secret "node-exporter-tls" not found
Sep 17 15:40:41.078252 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.078230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4646ca52-be14-4602-b081-160d471768ec-node-exporter-accelerators-collector-config\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.079947 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.079925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4646ca52-be14-4602-b081-160d471768ec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.087140 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.087121    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zlnk\" (UniqueName: \"kubernetes.io/projected/4646ca52-be14-4602-b081-160d471768ec-kube-api-access-4zlnk\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.362914 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.362836    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w"]
Sep 17 15:40:41.365565 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:41.365542    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e095828_1ae8_4785_be45_76814c0d11fb.slice/crio-9503fbd27a8f8eb77893bf4b318f9a2f22f5c79124600a793ee2b986c47e4b41 WatchSource:0}: Error finding container 9503fbd27a8f8eb77893bf4b318f9a2f22f5c79124600a793ee2b986c47e4b41: Status 404 returned error can't find the container with id 9503fbd27a8f8eb77893bf4b318f9a2f22f5c79124600a793ee2b986c47e4b41
Sep 17 15:40:41.581672 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.581632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4646ca52-be14-4602-b081-160d471768ec-node-exporter-tls\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.583956 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.583938    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4646ca52-be14-4602-b081-160d471768ec-node-exporter-tls\") pod \"node-exporter-s8bwq\" (UID: \"4646ca52-be14-4602-b081-160d471768ec\") " pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.693858 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.693781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s8bwq"
Sep 17 15:40:41.705086 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:41.705059    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4646ca52_be14_4602_b081_160d471768ec.slice/crio-9a8dffd35d28f44f7fd26687cc29d352689f88e8873806878cfe7d11e2414715 WatchSource:0}: Error finding container 9a8dffd35d28f44f7fd26687cc29d352689f88e8873806878cfe7d11e2414715: Status 404 returned error can't find the container with id 9a8dffd35d28f44f7fd26687cc29d352689f88e8873806878cfe7d11e2414715
Sep 17 15:40:41.828120 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.828085    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:40:41.832982 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.832959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.834867 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.834846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:40:41.834992 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.834847    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:40:41.834992 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.834927    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:40:41.835154 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.835137    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wrnlf\""
Sep 17 15:40:41.835213 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.835196    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:40:41.835254 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.835226    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:40:41.835313 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.835295    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:40:41.835406 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.835393    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:40:41.835457 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.835403    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:40:41.835457 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.835394    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:40:41.847274 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.847249    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:40:41.974180 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.974088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8bwq" event={"ID":"4646ca52-be14-4602-b081-160d471768ec","Type":"ContainerStarted","Data":"9a8dffd35d28f44f7fd26687cc29d352689f88e8873806878cfe7d11e2414715"}
Sep 17 15:40:41.975816 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.975788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w" event={"ID":"7e095828-1ae8-4785-be45-76814c0d11fb","Type":"ContainerStarted","Data":"adb50fb7f2a9c0fb643aed27690afd382aa5df38cf1f19147b705496b3db1553"}
Sep 17 15:40:41.975921 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.975851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w" event={"ID":"7e095828-1ae8-4785-be45-76814c0d11fb","Type":"ContainerStarted","Data":"80cd1683776661c40fcda9853b7f7629b080ddccb166bb2694803e4f8cf11f6c"}
Sep 17 15:40:41.975921 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.975881    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w" event={"ID":"7e095828-1ae8-4785-be45-76814c0d11fb","Type":"ContainerStarted","Data":"9503fbd27a8f8eb77893bf4b318f9a2f22f5c79124600a793ee2b986c47e4b41"}
Sep 17 15:40:41.984515 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984662 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-web-config\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984662 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qdf4\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-kube-api-access-2qdf4\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984662 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-volume\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984814 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984814 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984724    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-out\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984814 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984814 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984998 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984831    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984998 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984900    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984998 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984924    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984998 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984942    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:41.984998 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:41.984957    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.085950 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.085900    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.085950 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.085949    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-web-config\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.085979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qdf4\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-kube-api-access-2qdf4\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-volume\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086059    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-out\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086615 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086262    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086615 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086615 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086615 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086336    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086816 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.086770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.086899 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:42.086876    2581 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 17 15:40:42.086960 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:42.086940    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-main-tls podName:46b23a53-2892-42d1-89ca-667c0aac3bfc nodeName:}" failed. No retries permitted until 2026-09-17 15:40:42.586920815 +0000 UTC m=+133.510495365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc") : secret "alertmanager-main-tls" not found
Sep 17 15:40:42.087885 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.087860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.088645 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.088618    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.090040 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.090017    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-out\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.090861 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.090700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.091416 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.091371    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.092044 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.092021    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-web-config\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.092633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.092615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-volume\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.092702 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.092640    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.093098 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.092965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.094035 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.094011    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.095565 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.095541    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qdf4\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-kube-api-access-2qdf4\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.590433 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.590388    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.593384 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.593320    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.743388 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.743366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:40:42.863153 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.863093    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"]
Sep 17 15:40:42.866350 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.866335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:42.869397 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.869346    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 15:40:42.869507 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.869388    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 15:40:42.869557 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.869526    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7kgfl5el6oihf\""
Sep 17 15:40:42.869733 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.869716    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 15:40:42.870072 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.870056    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-q7cd8\""
Sep 17 15:40:42.870176 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.870158    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 15:40:42.870238 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.870201    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 15:40:42.887046 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.887022    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"]
Sep 17 15:40:42.986414 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.986377    2581 generic.go:356] "Generic (PLEG): container finished" podID="4646ca52-be14-4602-b081-160d471768ec" containerID="9878222fa4831d77a07f172128918329ebc14b2c9cfa69f88746f3d6fc82abd5" exitCode=0
Sep 17 15:40:42.986590 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.986476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8bwq" event={"ID":"4646ca52-be14-4602-b081-160d471768ec","Type":"ContainerDied","Data":"9878222fa4831d77a07f172128918329ebc14b2c9cfa69f88746f3d6fc82abd5"}
Sep 17 15:40:42.988762 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.988740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w" event={"ID":"7e095828-1ae8-4785-be45-76814c0d11fb","Type":"ContainerStarted","Data":"295a44c7d076e1b5ec2825a528744adc7d937992fc0452cdb1623c6e96a9941a"}
Sep 17 15:40:42.994310 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.994289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-grpc-tls\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:42.994403 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.994322    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:42.994403 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.994341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:42.994403 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.994375    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:42.994514 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.994421    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0dd1976-101f-4941-a5c2-2341bcddb7ea-metrics-client-ca\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:42.994514 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.994457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdvht\" (UniqueName: \"kubernetes.io/projected/d0dd1976-101f-4941-a5c2-2341bcddb7ea-kube-api-access-bdvht\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:42.994618 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.994533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-tls\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:42.994618 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:42.994562    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.020199 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.020154    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-9bp6w" podStartSLOduration=1.8047586020000002 podStartE2EDuration="3.0201381s" totalImagesPullingTime="1.215379498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:43.019035636 +0000 UTC m=+133.942610215" watchObservedRunningTime="2026-09-17 15:40:43.0201381 +0000 UTC m=+133.943712667"
Sep 17 15:40:43.033590 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.033532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:40:43.040373 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:43.040353    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46b23a53_2892_42d1_89ca_667c0aac3bfc.slice/crio-b59c3f86ebdcb03de36e417f9422b55ebaf2ec4f6da3e73817b39d7797afb070 WatchSource:0}: Error finding container b59c3f86ebdcb03de36e417f9422b55ebaf2ec4f6da3e73817b39d7797afb070: Status 404 returned error can't find the container with id b59c3f86ebdcb03de36e417f9422b55ebaf2ec4f6da3e73817b39d7797afb070
Sep 17 15:40:43.094925 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.094903    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0dd1976-101f-4941-a5c2-2341bcddb7ea-metrics-client-ca\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.095059 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.095037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdvht\" (UniqueName: \"kubernetes.io/projected/d0dd1976-101f-4941-a5c2-2341bcddb7ea-kube-api-access-bdvht\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.095140 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.095118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-tls\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.095354 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.095324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.095574 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.095555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-grpc-tls\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.095669 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.095624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.095732 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.095674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.095732 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.095717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d0dd1976-101f-4941-a5c2-2341bcddb7ea-metrics-client-ca\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.095833 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.095707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.098162 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.098138    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.098579 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.098490    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-tls\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.098933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.098914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.099037 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.098963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.099088 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.099048    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-grpc-tls\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.099088 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.099066    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d0dd1976-101f-4941-a5c2-2341bcddb7ea-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.102239 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.102219    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdvht\" (UniqueName: \"kubernetes.io/projected/d0dd1976-101f-4941-a5c2-2341bcddb7ea-kube-api-access-bdvht\") pod \"thanos-querier-5557b7d44b-2jfp6\" (UID: \"d0dd1976-101f-4941-a5c2-2341bcddb7ea\") " pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.197012 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.196923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:43.470724 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.470701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"]
Sep 17 15:40:43.472994 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:43.472952    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0dd1976_101f_4941_a5c2_2341bcddb7ea.slice/crio-b74374f7474501893d6762b0b8eca6c3bb06222c17bfc3bf379b48f367fc364b WatchSource:0}: Error finding container b74374f7474501893d6762b0b8eca6c3bb06222c17bfc3bf379b48f367fc364b: Status 404 returned error can't find the container with id b74374f7474501893d6762b0b8eca6c3bb06222c17bfc3bf379b48f367fc364b
Sep 17 15:40:43.994529 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.994495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8bwq" event={"ID":"4646ca52-be14-4602-b081-160d471768ec","Type":"ContainerStarted","Data":"98827ee271544debfbdb52907d35fd86a4c9976d177792287d56de675fea28be"}
Sep 17 15:40:43.994917 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.994559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s8bwq" event={"ID":"4646ca52-be14-4602-b081-160d471768ec","Type":"ContainerStarted","Data":"76d4b493129da5548713f30f786b1df4757b563b2267cff0f1c4475a1160c1e9"}
Sep 17 15:40:43.995678 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.995659    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6" event={"ID":"d0dd1976-101f-4941-a5c2-2341bcddb7ea","Type":"ContainerStarted","Data":"b74374f7474501893d6762b0b8eca6c3bb06222c17bfc3bf379b48f367fc364b"}
Sep 17 15:40:43.996726 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:43.996700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerStarted","Data":"b59c3f86ebdcb03de36e417f9422b55ebaf2ec4f6da3e73817b39d7797afb070"}
Sep 17 15:40:44.018152 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:44.018032    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s8bwq" podStartSLOduration=2.946312227 podStartE2EDuration="4.018011183s" totalImagesPullingTime="998.687237ms" totalInitContainerRuntime="73.011719ms" isStatefulPod=true podCreationTimestamp="2026-09-17 15:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:44.01665612 +0000 UTC m=+134.940230721" watchObservedRunningTime="2026-09-17 15:40:44.018011183 +0000 UTC m=+134.941585752"
Sep 17 15:40:45.003018 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.002980    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerID="a288937844afd779fe137fc15348f69fa75ea0a043335173935d3abbaf5c3171" exitCode=0
Sep 17 15:40:45.003401 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.003070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerDied","Data":"a288937844afd779fe137fc15348f69fa75ea0a043335173935d3abbaf5c3171"}
Sep 17 15:40:45.529292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.529265    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"]
Sep 17 15:40:45.534682 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.534657    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:45.536450 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.536429    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 17 15:40:45.536681 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.536648    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nzvrp\""
Sep 17 15:40:45.540146 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.540125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"]
Sep 17 15:40:45.619796 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.619725    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlx75\" (UniqueName: \"kubernetes.io/projected/ee283caa-b59d-4122-a21d-80ef64e92fb7-kube-api-access-mlx75\") pod \"monitoring-plugin-5d85778b9d-fs6mh\" (UID: \"ee283caa-b59d-4122-a21d-80ef64e92fb7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:45.619937 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.619890    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ee283caa-b59d-4122-a21d-80ef64e92fb7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fs6mh\" (UID: \"ee283caa-b59d-4122-a21d-80ef64e92fb7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:45.721182 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.721149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ee283caa-b59d-4122-a21d-80ef64e92fb7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fs6mh\" (UID: \"ee283caa-b59d-4122-a21d-80ef64e92fb7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:45.721377 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.721205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlx75\" (UniqueName: \"kubernetes.io/projected/ee283caa-b59d-4122-a21d-80ef64e92fb7-kube-api-access-mlx75\") pod \"monitoring-plugin-5d85778b9d-fs6mh\" (UID: \"ee283caa-b59d-4122-a21d-80ef64e92fb7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:45.721377 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:45.721327    2581 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 17 15:40:45.721522 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:45.721404    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee283caa-b59d-4122-a21d-80ef64e92fb7-monitoring-plugin-cert podName:ee283caa-b59d-4122-a21d-80ef64e92fb7 nodeName:}" failed. No retries permitted until 2026-09-17 15:40:46.221384531 +0000 UTC m=+137.144959093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/ee283caa-b59d-4122-a21d-80ef64e92fb7-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-fs6mh" (UID: "ee283caa-b59d-4122-a21d-80ef64e92fb7") : secret "monitoring-plugin-cert" not found
Sep 17 15:40:45.730277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:45.730253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlx75\" (UniqueName: \"kubernetes.io/projected/ee283caa-b59d-4122-a21d-80ef64e92fb7-kube-api-access-mlx75\") pod \"monitoring-plugin-5d85778b9d-fs6mh\" (UID: \"ee283caa-b59d-4122-a21d-80ef64e92fb7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:46.010307 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:46.010222    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6" event={"ID":"d0dd1976-101f-4941-a5c2-2341bcddb7ea","Type":"ContainerStarted","Data":"e13b97b8deef1f38e6cdd1b495bb4c31729e3d9c0438cbff64e421edf3b196d8"}
Sep 17 15:40:46.010816 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:46.010310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6" event={"ID":"d0dd1976-101f-4941-a5c2-2341bcddb7ea","Type":"ContainerStarted","Data":"c8adabb4515245a9f811b0e9b23083b27dfec35c8d2427f39ee353f4dc9b3d35"}
Sep 17 15:40:46.010816 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:46.010355    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6" event={"ID":"d0dd1976-101f-4941-a5c2-2341bcddb7ea","Type":"ContainerStarted","Data":"bba1164738c0aca2a0c1ce8185b35e4e47528e42e0b5b58db7bb96a245bf4599"}
Sep 17 15:40:46.225351 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:46.225316    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ee283caa-b59d-4122-a21d-80ef64e92fb7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fs6mh\" (UID: \"ee283caa-b59d-4122-a21d-80ef64e92fb7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:46.227592 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:46.227564    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ee283caa-b59d-4122-a21d-80ef64e92fb7-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-fs6mh\" (UID: \"ee283caa-b59d-4122-a21d-80ef64e92fb7\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:46.456186 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:46.456158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:46.741978 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:46.741824    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"]
Sep 17 15:40:46.744253 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:46.744229    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee283caa_b59d_4122_a21d_80ef64e92fb7.slice/crio-3b14cdf3f1e6819b15fe9eae6b658694d88e4085d5613abbd9f11c6d52599a22 WatchSource:0}: Error finding container 3b14cdf3f1e6819b15fe9eae6b658694d88e4085d5613abbd9f11c6d52599a22: Status 404 returned error can't find the container with id 3b14cdf3f1e6819b15fe9eae6b658694d88e4085d5613abbd9f11c6d52599a22
Sep 17 15:40:46.997938 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:46.997908    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:40:47.002039 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.002009    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.005095 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.004714    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 15:40:47.005095 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.004721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 15:40:47.005095 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.004940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 15:40:47.005095 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.004998    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 15:40:47.005363 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.005120    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:40:47.007006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.005422    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 15:40:47.007006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.005549    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 15:40:47.007006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.005809    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 15:40:47.007006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.005883    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 15:40:47.007006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.006175    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 15:40:47.007006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.006216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 15:40:47.007006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.006644    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bhec0mvvuq16b\""
Sep 17 15:40:47.007006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.006738    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pr68z\""
Sep 17 15:40:47.007006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.006957    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 15:40:47.007541 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.007407    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 15:40:47.008521 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.008500    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 15:40:47.013783 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.013604    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 15:40:47.027403 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.024109    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:40:47.029506 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.029454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6" event={"ID":"d0dd1976-101f-4941-a5c2-2341bcddb7ea","Type":"ContainerStarted","Data":"8bd82aa6d330d8627ae0ced97f5e82c9196e8f51958e4bd58e5c7f9ac8427377"}
Sep 17 15:40:47.029746 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.029728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:47.029942 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.029922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6" event={"ID":"d0dd1976-101f-4941-a5c2-2341bcddb7ea","Type":"ContainerStarted","Data":"f5dbd0776fb604b01680f019fe19be958aab9ee1ed869a4cb557ee8d1d2e1188"}
Sep 17 15:40:47.030103 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.030088    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6" event={"ID":"d0dd1976-101f-4941-a5c2-2341bcddb7ea","Type":"ContainerStarted","Data":"ce653df07d9e11dcefd27719ba2d70a7722332da0d6127232177cb07e1e38fa2"}
Sep 17 15:40:47.033930 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.033794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerStarted","Data":"800ee94aa15f638eb00b2ba21316a0abdac7d703d92d71b4fe66da9502c80084"}
Sep 17 15:40:47.033930 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.033864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerStarted","Data":"dd00805d1b85b69089c71830760e96f464b2cf87f24d74f6fe014f4bdc135eb9"}
Sep 17 15:40:47.033930 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.033900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerStarted","Data":"23ec83c017fb201a49c5b937203214334a81346eee110ea88040611f6a5ab1c3"}
Sep 17 15:40:47.033930 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.033935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerStarted","Data":"c8c19636ae56d3361202ab918baaa41ddb92879a7351dca7c1711a759a757cfb"}
Sep 17 15:40:47.034168 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.033970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerStarted","Data":"17631d06ddc85c65c515941c8a173f465238f991707beac2876f8c77f8806161"}
Sep 17 15:40:47.034730 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.034705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.034940 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.034902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.035085 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.035067    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.035206 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.035190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.035321 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.035304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.035421 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.035373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh" event={"ID":"ee283caa-b59d-4122-a21d-80ef64e92fb7","Type":"ContainerStarted","Data":"3b14cdf3f1e6819b15fe9eae6b658694d88e4085d5613abbd9f11c6d52599a22"}
Sep 17 15:40:47.035558 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.035540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.035661 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.035646    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.035774 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.035757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.035896 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.035881    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.036086 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.036069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.036199 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.036184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.036303 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.036289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.036404 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.036390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.036532 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.036516    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.036652 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.036637    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.036771 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.036755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.036891 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.036877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.036990 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.036975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.037083 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.037070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.037195 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.037181    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcqz7\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-kube-api-access-pcqz7\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.078572 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.078516    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6" podStartSLOduration=1.983648973 podStartE2EDuration="5.078494263s" totalImagesPullingTime="3.09484529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:47.078329927 +0000 UTC m=+138.001904495" watchObservedRunningTime="2026-09-17 15:40:47.078494263 +0000 UTC m=+138.002068832"
Sep 17 15:40:47.138243 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138212    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138392 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138392 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138392 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138392 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138599 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138578    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138638 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138693 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138757 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138792 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138778    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138834 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138803    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138887 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138877    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcqz7\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-kube-api-access-pcqz7\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138920    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.138985 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.139038 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.138984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.139589 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.139562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.141977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.139773    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.141977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.140365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.141977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.140414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.141977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.140455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.141977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.140515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.141977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.140544    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.141977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.140588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.141977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.140764    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.141977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.141163    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.144442 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.144418    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.144713 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.144422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.144713 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.144496    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.145025 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.144985    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.145363 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.145341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.145547 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.145527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.145627 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.145607    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-web-config\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.145916 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.145890    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.145996 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.145960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.146186 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.146168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.146287 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.146239    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.146353 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.146339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config-out\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.146685 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.146655    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.146783 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.146763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.148595 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.148579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcqz7\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-kube-api-access-pcqz7\") pod \"prometheus-k8s-0\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.315340 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.315260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:47.612177 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:47.612151    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:40:47.854519 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:40:47.854485    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f0cd02_3d74_41ca_8efc_fb55017af6c4.slice/crio-12df9f40d4292c2fadf77b8fc146480a7f1e5b7942910acb0cf11de12052be0d WatchSource:0}: Error finding container 12df9f40d4292c2fadf77b8fc146480a7f1e5b7942910acb0cf11de12052be0d: Status 404 returned error can't find the container with id 12df9f40d4292c2fadf77b8fc146480a7f1e5b7942910acb0cf11de12052be0d
Sep 17 15:40:48.040880 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:48.040850    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerID="46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb" exitCode=0
Sep 17 15:40:48.041338 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:48.040932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerDied","Data":"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"}
Sep 17 15:40:48.041338 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:48.040999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerStarted","Data":"12df9f40d4292c2fadf77b8fc146480a7f1e5b7942910acb0cf11de12052be0d"}
Sep 17 15:40:48.044649 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:48.044626    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerStarted","Data":"fa2feb439e6d8e42c247db9677c9196ae1be3fc59f86800ee58c77686d7c6f18"}
Sep 17 15:40:48.046238 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:48.046215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh" event={"ID":"ee283caa-b59d-4122-a21d-80ef64e92fb7","Type":"ContainerStarted","Data":"5e9ab238deffcd256bf3af78c1f9fb85bc5a89cab4638b67e813223926b491c2"}
Sep 17 15:40:48.046563 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:48.046544    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:48.051292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:48.051268    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh"
Sep 17 15:40:48.090921 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:48.090855    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.402098013 podStartE2EDuration="7.090834364s" totalImagesPullingTime="2.672498487s" totalInitContainerRuntime="16.237864ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:48.090375938 +0000 UTC m=+139.013950497" watchObservedRunningTime="2026-09-17 15:40:48.090834364 +0000 UTC m=+139.014408934"
Sep 17 15:40:48.107042 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:48.106984    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-fs6mh" podStartSLOduration=1.948586826 podStartE2EDuration="3.106967897s" totalImagesPullingTime="1.158381071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:48.106524039 +0000 UTC m=+139.030098606" watchObservedRunningTime="2026-09-17 15:40:48.106967897 +0000 UTC m=+139.030542475"
Sep 17 15:40:52.066582 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:52.066543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerStarted","Data":"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"}
Sep 17 15:40:52.067007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:52.066658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerStarted","Data":"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"}
Sep 17 15:40:52.067007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:52.066703    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerStarted","Data":"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"}
Sep 17 15:40:52.067007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:52.066746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerStarted","Data":"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"}
Sep 17 15:40:52.067007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:52.066781    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerStarted","Data":"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"}
Sep 17 15:40:52.067007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:52.066819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerStarted","Data":"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"}
Sep 17 15:40:52.104769 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:52.104712    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7963120569999997 podStartE2EDuration="6.104691135s" totalImagesPullingTime="3.285504034s" totalInitContainerRuntime="22.875044ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:40:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:40:52.104296219 +0000 UTC m=+143.027870785" watchObservedRunningTime="2026-09-17 15:40:52.104691135 +0000 UTC m=+143.028265705"
Sep 17 15:40:52.316064 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:52.316026    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:40:54.056500 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:54.056449    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5557b7d44b-2jfp6"
Sep 17 15:40:56.314005 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:56.313961    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b444dd987-vqlxd"]
Sep 17 15:40:56.314494 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:40:56.314329    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-7b444dd987-vqlxd" podUID="6455c538-28c2-4ae8-957f-eea968f8ce7a"
Sep 17 15:40:57.086658 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.086627    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:40:57.094727 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.094701    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:40:57.134407 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.134371    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-certificates\") pod \"6455c538-28c2-4ae8-957f-eea968f8ce7a\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") "
Sep 17 15:40:57.134407 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.134420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-installation-pull-secrets\") pod \"6455c538-28c2-4ae8-957f-eea968f8ce7a\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") "
Sep 17 15:40:57.134611 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.134452    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj47s\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-kube-api-access-cj47s\") pod \"6455c538-28c2-4ae8-957f-eea968f8ce7a\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") "
Sep 17 15:40:57.134611 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.134512    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-bound-sa-token\") pod \"6455c538-28c2-4ae8-957f-eea968f8ce7a\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") "
Sep 17 15:40:57.134611 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.134535    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-trusted-ca\") pod \"6455c538-28c2-4ae8-957f-eea968f8ce7a\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") "
Sep 17 15:40:57.134771 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.134620    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-image-registry-private-configuration\") pod \"6455c538-28c2-4ae8-957f-eea968f8ce7a\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") "
Sep 17 15:40:57.134771 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.134675    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455c538-28c2-4ae8-957f-eea968f8ce7a-ca-trust-extracted\") pod \"6455c538-28c2-4ae8-957f-eea968f8ce7a\" (UID: \"6455c538-28c2-4ae8-957f-eea968f8ce7a\") "
Sep 17 15:40:57.134870 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.134759    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "6455c538-28c2-4ae8-957f-eea968f8ce7a" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:40:57.134987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.134964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6455c538-28c2-4ae8-957f-eea968f8ce7a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "6455c538-28c2-4ae8-957f-eea968f8ce7a" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:40:57.135061 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.135028    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "6455c538-28c2-4ae8-957f-eea968f8ce7a" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:40:57.135061 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.135039    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-certificates\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:40:57.136880 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.136852    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "6455c538-28c2-4ae8-957f-eea968f8ce7a" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:40:57.136973 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.136877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-kube-api-access-cj47s" (OuterVolumeSpecName: "kube-api-access-cj47s") pod "6455c538-28c2-4ae8-957f-eea968f8ce7a" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a"). InnerVolumeSpecName "kube-api-access-cj47s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:40:57.137133 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.137113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "6455c538-28c2-4ae8-957f-eea968f8ce7a" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:40:57.137187 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.137136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "6455c538-28c2-4ae8-957f-eea968f8ce7a" (UID: "6455c538-28c2-4ae8-957f-eea968f8ce7a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:40:57.236384 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.236341    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj47s\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-kube-api-access-cj47s\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:40:57.236384 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.236374    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-bound-sa-token\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:40:57.236384 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.236385    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6455c538-28c2-4ae8-957f-eea968f8ce7a-trusted-ca\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:40:57.236384 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.236394    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-image-registry-private-configuration\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:40:57.236661 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.236403    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6455c538-28c2-4ae8-957f-eea968f8ce7a-ca-trust-extracted\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:40:57.236661 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:57.236413    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6455c538-28c2-4ae8-957f-eea968f8ce7a-installation-pull-secrets\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:40:58.090623 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:58.090593    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b444dd987-vqlxd"
Sep 17 15:40:58.125879 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:58.125846    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b444dd987-vqlxd"]
Sep 17 15:40:58.135339 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:58.135311    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b444dd987-vqlxd"]
Sep 17 15:40:58.243217 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:58.243190    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6455c538-28c2-4ae8-957f-eea968f8ce7a-registry-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:40:59.561382 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:40:59.561349    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6455c538-28c2-4ae8-957f-eea968f8ce7a" path="/var/lib/kubelet/pods/6455c538-28c2-4ae8-957f-eea968f8ce7a/volumes"
Sep 17 15:41:04.918069 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:41:04.918008    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-sfthf" podUID="0dc7d157-3364-4dde-93a0-06e97da1db07"
Sep 17 15:41:05.116669 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:05.116641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sfthf"
Sep 17 15:41:09.838940 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:09.838888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:41:09.841284 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:09.841262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dc7d157-3364-4dde-93a0-06e97da1db07-metrics-tls\") pod \"dns-default-sfthf\" (UID: \"0dc7d157-3364-4dde-93a0-06e97da1db07\") " pod="openshift-dns/dns-default-sfthf"
Sep 17 15:41:09.919715 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:09.919682    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ghpsh\""
Sep 17 15:41:09.928356 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:09.928335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sfthf"
Sep 17 15:41:10.200759 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:10.200689    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sfthf"]
Sep 17 15:41:10.204352 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:41:10.204324    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dc7d157_3364_4dde_93a0_06e97da1db07.slice/crio-88d105b0e8e22d3491feb7e321f1d5b470e3cca9a4a726457ca7d0b67e07b2dd WatchSource:0}: Error finding container 88d105b0e8e22d3491feb7e321f1d5b470e3cca9a4a726457ca7d0b67e07b2dd: Status 404 returned error can't find the container with id 88d105b0e8e22d3491feb7e321f1d5b470e3cca9a4a726457ca7d0b67e07b2dd
Sep 17 15:41:11.143574 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:11.143529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sfthf" event={"ID":"0dc7d157-3364-4dde-93a0-06e97da1db07","Type":"ContainerStarted","Data":"88d105b0e8e22d3491feb7e321f1d5b470e3cca9a4a726457ca7d0b67e07b2dd"}
Sep 17 15:41:12.149628 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:12.149542    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sfthf" event={"ID":"0dc7d157-3364-4dde-93a0-06e97da1db07","Type":"ContainerStarted","Data":"d3f28ad80b5c568c95ea5bcdc8b7ee6d2672898a713fe8895429fc5f14ca938f"}
Sep 17 15:41:12.149999 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:12.149684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sfthf" event={"ID":"0dc7d157-3364-4dde-93a0-06e97da1db07","Type":"ContainerStarted","Data":"93e095d512aad3f3e4bf8f344260052a75c245b856cded18036af8077b9e19ee"}
Sep 17 15:41:12.149999 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:12.149717    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sfthf"
Sep 17 15:41:12.167256 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:12.167205    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sfthf" podStartSLOduration=129.679341398 podStartE2EDuration="2m11.167191045s" totalImagesPullingTime="1.487849647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:41:12.1662399 +0000 UTC m=+163.089814461" watchObservedRunningTime="2026-09-17 15:41:12.167191045 +0000 UTC m=+163.090765641"
Sep 17 15:41:22.155750 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:22.155669    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sfthf"
Sep 17 15:41:24.201402 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:24.201376    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sfthf_0dc7d157-3364-4dde-93a0-06e97da1db07/dns/0.log"
Sep 17 15:41:24.401275 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:24.401248    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sfthf_0dc7d157-3364-4dde-93a0-06e97da1db07/kube-rbac-proxy/0.log"
Sep 17 15:41:24.601054 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:24.601026    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7gbxk_d033245e-d93b-4b38-b1c8-62351fd7cfc1/dns-node-resolver/0.log"
Sep 17 15:41:25.201447 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:25.201420    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-648db6667b-2h95h_b4db3ce1-346c-42de-8ca6-240881068330/router/0.log"
Sep 17 15:41:25.603691 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:25.603661    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jdr9w_d25033f9-7df1-47bb-a41a-3e3a57d36da5/serve-healthcheck-canary/0.log"
Sep 17 15:41:47.315734 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:47.315701    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:47.330938 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:47.330913    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:41:48.300992 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:41:48.300965    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:01.124281 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.124244    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:01.124836 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.124626    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="alertmanager" containerID="cri-o://17631d06ddc85c65c515941c8a173f465238f991707beac2876f8c77f8806161" gracePeriod=120
Sep 17 15:42:01.124836 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.124721    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy-metric" containerID="cri-o://800ee94aa15f638eb00b2ba21316a0abdac7d703d92d71b4fe66da9502c80084" gracePeriod=120
Sep 17 15:42:01.124969 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.124812    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy-web" containerID="cri-o://23ec83c017fb201a49c5b937203214334a81346eee110ea88040611f6a5ab1c3" gracePeriod=120
Sep 17 15:42:01.124969 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.124865    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy" containerID="cri-o://dd00805d1b85b69089c71830760e96f464b2cf87f24d74f6fe014f4bdc135eb9" gracePeriod=120
Sep 17 15:42:01.124969 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.124807    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="prom-label-proxy" containerID="cri-o://fa2feb439e6d8e42c247db9677c9196ae1be3fc59f86800ee58c77686d7c6f18" gracePeriod=120
Sep 17 15:42:01.124969 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.124868    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="config-reloader" containerID="cri-o://c8c19636ae56d3361202ab918baaa41ddb92879a7351dca7c1711a759a757cfb" gracePeriod=120
Sep 17 15:42:01.337846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.337817    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerID="fa2feb439e6d8e42c247db9677c9196ae1be3fc59f86800ee58c77686d7c6f18" exitCode=0
Sep 17 15:42:01.337846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.337841    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerID="dd00805d1b85b69089c71830760e96f464b2cf87f24d74f6fe014f4bdc135eb9" exitCode=0
Sep 17 15:42:01.337846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.337848    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerID="c8c19636ae56d3361202ab918baaa41ddb92879a7351dca7c1711a759a757cfb" exitCode=0
Sep 17 15:42:01.337846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.337853    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerID="17631d06ddc85c65c515941c8a173f465238f991707beac2876f8c77f8806161" exitCode=0
Sep 17 15:42:01.338108 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.337891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerDied","Data":"fa2feb439e6d8e42c247db9677c9196ae1be3fc59f86800ee58c77686d7c6f18"}
Sep 17 15:42:01.338108 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.337952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerDied","Data":"dd00805d1b85b69089c71830760e96f464b2cf87f24d74f6fe014f4bdc135eb9"}
Sep 17 15:42:01.338108 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.337979    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerDied","Data":"c8c19636ae56d3361202ab918baaa41ddb92879a7351dca7c1711a759a757cfb"}
Sep 17 15:42:01.338108 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:01.338003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerDied","Data":"17631d06ddc85c65c515941c8a173f465238f991707beac2876f8c77f8806161"}
Sep 17 15:42:02.345447 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.345420    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerID="800ee94aa15f638eb00b2ba21316a0abdac7d703d92d71b4fe66da9502c80084" exitCode=0
Sep 17 15:42:02.345447 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.345443    2581 generic.go:356] "Generic (PLEG): container finished" podID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerID="23ec83c017fb201a49c5b937203214334a81346eee110ea88040611f6a5ab1c3" exitCode=0
Sep 17 15:42:02.345845 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.345525    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerDied","Data":"800ee94aa15f638eb00b2ba21316a0abdac7d703d92d71b4fe66da9502c80084"}
Sep 17 15:42:02.345845 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.345610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerDied","Data":"23ec83c017fb201a49c5b937203214334a81346eee110ea88040611f6a5ab1c3"}
Sep 17 15:42:02.370790 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.370765    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:02.478433 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478347    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-web-config\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478433 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478410    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qdf4\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-kube-api-access-2qdf4\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478433 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478428    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-volume\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478684 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478444    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478684 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-main-tls\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478684 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478539    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-web\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478684 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478557    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-out\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478684 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478573    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-cluster-tls-config\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478941 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-trusted-ca-bundle\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478997 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478951    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-main-db\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.478997 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.478980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-tls-assets\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.479095 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.479019    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.479095 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.479063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-metrics-client-ca\") pod \"46b23a53-2892-42d1-89ca-667c0aac3bfc\" (UID: \"46b23a53-2892-42d1-89ca-667c0aac3bfc\") "
Sep 17 15:42:02.479953 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.479723    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:02.481443 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.481415    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-out" (OuterVolumeSpecName: "config-out") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:02.481798 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.481722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:02.481907 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.481826    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.481977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.481874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.482041 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.482015    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-kube-api-access-2qdf4" (OuterVolumeSpecName: "kube-api-access-2qdf4") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "kube-api-access-2qdf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:02.482101 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.482071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-volume" (OuterVolumeSpecName: "config-volume") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.482184 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.482152    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.482247 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.482229    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:02.484006 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.483973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:02.484128 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.484111    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.486156 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.486012    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.491678 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.491657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-web-config" (OuterVolumeSpecName: "web-config") pod "46b23a53-2892-42d1-89ca-667c0aac3bfc" (UID: "46b23a53-2892-42d1-89ca-667c0aac3bfc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:02.580292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580245    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qdf4\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-kube-api-access-2qdf4\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580287    2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-volume\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580298    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580308    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-main-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580318    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580327    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-config-out\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580335    2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-cluster-tls-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580343    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580352    2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/46b23a53-2892-42d1-89ca-667c0aac3bfc-alertmanager-main-db\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580359    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/46b23a53-2892-42d1-89ca-667c0aac3bfc-tls-assets\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580368    2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580377    2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46b23a53-2892-42d1-89ca-667c0aac3bfc-metrics-client-ca\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:02.580571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:02.580385    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/46b23a53-2892-42d1-89ca-667c0aac3bfc-web-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:03.352347 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.352312    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"46b23a53-2892-42d1-89ca-667c0aac3bfc","Type":"ContainerDied","Data":"b59c3f86ebdcb03de36e417f9422b55ebaf2ec4f6da3e73817b39d7797afb070"}
Sep 17 15:42:03.352796 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.352377    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.352796 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.352393    2581 scope.go:117] "RemoveContainer" containerID="fa2feb439e6d8e42c247db9677c9196ae1be3fc59f86800ee58c77686d7c6f18"
Sep 17 15:42:03.361128 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.361056    2581 scope.go:117] "RemoveContainer" containerID="800ee94aa15f638eb00b2ba21316a0abdac7d703d92d71b4fe66da9502c80084"
Sep 17 15:42:03.367904 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.367886    2581 scope.go:117] "RemoveContainer" containerID="dd00805d1b85b69089c71830760e96f464b2cf87f24d74f6fe014f4bdc135eb9"
Sep 17 15:42:03.374905 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.374883    2581 scope.go:117] "RemoveContainer" containerID="23ec83c017fb201a49c5b937203214334a81346eee110ea88040611f6a5ab1c3"
Sep 17 15:42:03.375971 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.375867    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:03.380225 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.380203    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:03.382745 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.382730    2581 scope.go:117] "RemoveContainer" containerID="c8c19636ae56d3361202ab918baaa41ddb92879a7351dca7c1711a759a757cfb"
Sep 17 15:42:03.389647 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.389621    2581 scope.go:117] "RemoveContainer" containerID="17631d06ddc85c65c515941c8a173f465238f991707beac2876f8c77f8806161"
Sep 17 15:42:03.396833 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.396816    2581 scope.go:117] "RemoveContainer" containerID="a288937844afd779fe137fc15348f69fa75ea0a043335173935d3abbaf5c3171"
Sep 17 15:42:03.404344 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404324    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:03.404640 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404627    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="alertmanager"
Sep 17 15:42:03.404701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404642    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="alertmanager"
Sep 17 15:42:03.404701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404652    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="config-reloader"
Sep 17 15:42:03.404701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404658    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="config-reloader"
Sep 17 15:42:03.404701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404664    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy-metric"
Sep 17 15:42:03.404701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404670    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy-metric"
Sep 17 15:42:03.404701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404683    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy"
Sep 17 15:42:03.404701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404688    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy"
Sep 17 15:42:03.404701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404697    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy-web"
Sep 17 15:42:03.404701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404702    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy-web"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404710    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="init-config-reloader"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404715    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="init-config-reloader"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404721    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="prom-label-proxy"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404727    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="prom-label-proxy"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404773    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="prom-label-proxy"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404782    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy-web"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404788    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="alertmanager"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404794    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="config-reloader"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404800    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy-metric"
Sep 17 15:42:03.404987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.404805    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" containerName="kube-rbac-proxy"
Sep 17 15:42:03.410179 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.410155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.411944 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.411920    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 15:42:03.412111 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.412093    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 15:42:03.412184 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.412113    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 15:42:03.412356 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.412344    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 15:42:03.412484 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.412441    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 15:42:03.412547 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.412455    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 15:42:03.412547 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.412445    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wrnlf\""
Sep 17 15:42:03.412547 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.412446    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 15:42:03.412662 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.412445    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 15:42:03.418743 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.418713    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 15:42:03.422242 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.422222    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:03.486572 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486572 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486572 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486572 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65313996-18e8-4205-b69a-2daa1d9cf68b-config-out\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486860 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/65313996-18e8-4205-b69a-2daa1d9cf68b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486860 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-web-config\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486860 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pgzp\" (UniqueName: \"kubernetes.io/projected/65313996-18e8-4205-b69a-2daa1d9cf68b-kube-api-access-6pgzp\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486860 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486860 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-config-volume\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486860 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65313996-18e8-4205-b69a-2daa1d9cf68b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486860 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486752    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65313996-18e8-4205-b69a-2daa1d9cf68b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486860 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486772    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65313996-18e8-4205-b69a-2daa1d9cf68b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.486860 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.486808    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.559539 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.559505    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46b23a53-2892-42d1-89ca-667c0aac3bfc" path="/var/lib/kubelet/pods/46b23a53-2892-42d1-89ca-667c0aac3bfc/volumes"
Sep 17 15:42:03.587199 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587281 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587211    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587281 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587281 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587397 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587280    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65313996-18e8-4205-b69a-2daa1d9cf68b-config-out\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587397 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/65313996-18e8-4205-b69a-2daa1d9cf68b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587397 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-web-config\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587397 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587341    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pgzp\" (UniqueName: \"kubernetes.io/projected/65313996-18e8-4205-b69a-2daa1d9cf68b-kube-api-access-6pgzp\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587397 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587674 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-config-volume\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587674 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65313996-18e8-4205-b69a-2daa1d9cf68b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587674 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65313996-18e8-4205-b69a-2daa1d9cf68b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.587674 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.587535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65313996-18e8-4205-b69a-2daa1d9cf68b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.588222 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.588190    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/65313996-18e8-4205-b69a-2daa1d9cf68b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.588851 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.588588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65313996-18e8-4205-b69a-2daa1d9cf68b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.589401 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.589377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/65313996-18e8-4205-b69a-2daa1d9cf68b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.590693 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.590129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/65313996-18e8-4205-b69a-2daa1d9cf68b-config-out\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.590693 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.590202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.590693 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.590616    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.590693 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.590648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.590945 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.590785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.591183 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.591160    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-web-config\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.591248 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.591208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-config-volume\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.591629 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.591610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/65313996-18e8-4205-b69a-2daa1d9cf68b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.591940 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.591922    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/65313996-18e8-4205-b69a-2daa1d9cf68b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.595691 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.595672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pgzp\" (UniqueName: \"kubernetes.io/projected/65313996-18e8-4205-b69a-2daa1d9cf68b-kube-api-access-6pgzp\") pod \"alertmanager-main-0\" (UID: \"65313996-18e8-4205-b69a-2daa1d9cf68b\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:03.724943 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:03.724907    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 15:42:04.026901 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:04.026818    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 15:42:04.030102 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:42:04.030079    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65313996_18e8_4205_b69a_2daa1d9cf68b.slice/crio-e53713633a4ad982f0b9ff8357d1d834a6601888150df21c07720514347f374a WatchSource:0}: Error finding container e53713633a4ad982f0b9ff8357d1d834a6601888150df21c07720514347f374a: Status 404 returned error can't find the container with id e53713633a4ad982f0b9ff8357d1d834a6601888150df21c07720514347f374a
Sep 17 15:42:04.357063 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:04.357033    2581 generic.go:356] "Generic (PLEG): container finished" podID="65313996-18e8-4205-b69a-2daa1d9cf68b" containerID="140f9b84d73821d216ec12b02091306e0baefdb6bad74b0b30ab594a7ae18c32" exitCode=0
Sep 17 15:42:04.357535 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:04.357129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65313996-18e8-4205-b69a-2daa1d9cf68b","Type":"ContainerDied","Data":"140f9b84d73821d216ec12b02091306e0baefdb6bad74b0b30ab594a7ae18c32"}
Sep 17 15:42:04.357535 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:04.357225    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65313996-18e8-4205-b69a-2daa1d9cf68b","Type":"ContainerStarted","Data":"e53713633a4ad982f0b9ff8357d1d834a6601888150df21c07720514347f374a"}
Sep 17 15:42:05.367086 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.367049    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65313996-18e8-4205-b69a-2daa1d9cf68b","Type":"ContainerStarted","Data":"1812a4fd6a4b82cce247dece9af88c3690c5183c43bbc2b6e4c06dc3d8167650"}
Sep 17 15:42:05.367581 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.367125    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65313996-18e8-4205-b69a-2daa1d9cf68b","Type":"ContainerStarted","Data":"b2b4b26a7d17fe3a315a173999dda5ae2c73e36fec1d29b8fbc098ff593d6194"}
Sep 17 15:42:05.367581 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.367165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65313996-18e8-4205-b69a-2daa1d9cf68b","Type":"ContainerStarted","Data":"08b72c58ea49f5d5c5da61bfb81f61e78d6bb73d785c00cb9e6c02912fb134e9"}
Sep 17 15:42:05.367581 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.367207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65313996-18e8-4205-b69a-2daa1d9cf68b","Type":"ContainerStarted","Data":"76162277a7dfbf3e919cbe7631b597ef82504e17c92184a1ca67d073cd4a61b1"}
Sep 17 15:42:05.367581 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.367255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65313996-18e8-4205-b69a-2daa1d9cf68b","Type":"ContainerStarted","Data":"9b797ec3d4c50910b1749c52b5139929755bdfc0a3098bf5d697c8a3ea3e991c"}
Sep 17 15:42:05.367581 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.367293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"65313996-18e8-4205-b69a-2daa1d9cf68b","Type":"ContainerStarted","Data":"75b74ccd4c3d80c5e6703307832908962ea6fd383488d969489d22569aa8bb64"}
Sep 17 15:42:05.371837 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.371811    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:05.372170 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.372137    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy" containerID="cri-o://574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6" gracePeriod=600
Sep 17 15:42:05.372283 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.372170    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764" gracePeriod=600
Sep 17 15:42:05.372283 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.372144    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy-web" containerID="cri-o://9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb" gracePeriod=600
Sep 17 15:42:05.372283 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.372144    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="thanos-sidecar" containerID="cri-o://38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff" gracePeriod=600
Sep 17 15:42:05.372437 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.372149    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="config-reloader" containerID="cri-o://b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c" gracePeriod=600
Sep 17 15:42:05.372437 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.372170    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="prometheus" containerID="cri-o://d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433" gracePeriod=600
Sep 17 15:42:05.400954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.400324    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.382210569 podStartE2EDuration="2.400303589s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.09302ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:05.398014051 +0000 UTC m=+216.321588621" watchObservedRunningTime="2026-09-17 15:42:05.400303589 +0000 UTC m=+216.323878157"
Sep 17 15:42:05.623648 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.623585    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:05.706680 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706648    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.706858 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-grpc-tls\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.706858 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-thanos-prometheus-http-client-file\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.706858 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706735    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config-out\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.706858 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706762    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.706858 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706794    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-trusted-ca-bundle\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.706858 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706818    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-kube-rbac-proxy\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.706858 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706843    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-metrics-client-ca\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706874    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-1\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706906    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-2\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706936    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-web-config\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.706970    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-serving-certs-ca-bundle\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcqz7\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-kube-api-access-pcqz7\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707035    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-tls\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707095    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-metrics-client-certs\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707128    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-kubelet-serving-ca-bundle\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707164    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-db\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-tls-assets\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707272    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-0\") pod \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\" (UID: \"d5f0cd02-3d74-41ca-8efc-fb55017af6c4\") "
Sep 17 15:42:05.707872 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707585    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:05.707935 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707890    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:05.708683 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.707990    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:05.708683 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.708147    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:05.708683 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.708271    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:05.708683 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.708291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:05.709014 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.708989    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:42:05.709625 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.709562    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:05.709886 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.709859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config-out" (OuterVolumeSpecName: "config-out") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:42:05.711572 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.711521    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-kube-api-access-pcqz7" (OuterVolumeSpecName: "kube-api-access-pcqz7") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "kube-api-access-pcqz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:05.711805 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.711742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:42:05.711805 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.711759    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:05.711805 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.711779    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config" (OuterVolumeSpecName: "config") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:05.711993 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.711811    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:05.711993 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.711877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:05.711993 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.711938    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:05.711993 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.711988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:05.712800 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.712777    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:05.713322 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.713295    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:05.722353 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.722329    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-web-config" (OuterVolumeSpecName: "web-config") pod "d5f0cd02-3d74-41ca-8efc-fb55017af6c4" (UID: "d5f0cd02-3d74-41ca-8efc-fb55017af6c4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:42:05.808586 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808553    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808586 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808585    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808596    2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-grpc-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808605    2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808615    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config-out\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808624    2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808631    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808639    2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-kube-rbac-proxy\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808648    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-metrics-client-ca\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808656    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808663    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808672    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-web-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808680    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808689    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcqz7\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-kube-api-access-pcqz7\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808698    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808707    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808715    2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-secret-metrics-client-certs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808723    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.808728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808731    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-prometheus-k8s-db\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:05.809194 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:05.808738    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5f0cd02-3d74-41ca-8efc-fb55017af6c4-tls-assets\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:42:06.374024 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.373991    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerID="e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764" exitCode=0
Sep 17 15:42:06.374024 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374014    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerID="574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6" exitCode=0
Sep 17 15:42:06.374024 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374021    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerID="9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb" exitCode=0
Sep 17 15:42:06.374024 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374026    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerID="38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff" exitCode=0
Sep 17 15:42:06.374024 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374032    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerID="b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c" exitCode=0
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374037    2581 generic.go:356] "Generic (PLEG): container finished" podID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerID="d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433" exitCode=0
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374064    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerDied","Data":"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"}
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374221    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerDied","Data":"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"}
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerDied","Data":"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"}
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerDied","Data":"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"}
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374282    2581 scope.go:117] "RemoveContainer" containerID="e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374298    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerDied","Data":"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"}
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerDied","Data":"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"}
Sep 17 15:42:06.374633 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.374356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d5f0cd02-3d74-41ca-8efc-fb55017af6c4","Type":"ContainerDied","Data":"12df9f40d4292c2fadf77b8fc146480a7f1e5b7942910acb0cf11de12052be0d"}
Sep 17 15:42:06.382648 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.382629    2581 scope.go:117] "RemoveContainer" containerID="574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"
Sep 17 15:42:06.389789 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.389768    2581 scope.go:117] "RemoveContainer" containerID="9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"
Sep 17 15:42:06.395908 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.395894    2581 scope.go:117] "RemoveContainer" containerID="38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"
Sep 17 15:42:06.399604 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.399581    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:06.403261 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.403240    2581 scope.go:117] "RemoveContainer" containerID="b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"
Sep 17 15:42:06.403355 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.403293    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:06.409724 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.409704    2581 scope.go:117] "RemoveContainer" containerID="d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"
Sep 17 15:42:06.416511 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.416496    2581 scope.go:117] "RemoveContainer" containerID="46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"
Sep 17 15:42:06.422817 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.422803    2581 scope.go:117] "RemoveContainer" containerID="e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"
Sep 17 15:42:06.423055 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:42:06.423036    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": container with ID starting with e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764 not found: ID does not exist" containerID="e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"
Sep 17 15:42:06.423114 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.423063    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"} err="failed to get container status \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": rpc error: code = NotFound desc = could not find container \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": container with ID starting with e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764 not found: ID does not exist"
Sep 17 15:42:06.423114 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.423092    2581 scope.go:117] "RemoveContainer" containerID="574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"
Sep 17 15:42:06.423328 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:42:06.423314    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": container with ID starting with 574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6 not found: ID does not exist" containerID="574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"
Sep 17 15:42:06.423363 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.423332    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"} err="failed to get container status \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": rpc error: code = NotFound desc = could not find container \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": container with ID starting with 574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6 not found: ID does not exist"
Sep 17 15:42:06.423363 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.423347    2581 scope.go:117] "RemoveContainer" containerID="9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"
Sep 17 15:42:06.423594 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:42:06.423579    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": container with ID starting with 9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb not found: ID does not exist" containerID="9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"
Sep 17 15:42:06.423642 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.423597    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"} err="failed to get container status \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": rpc error: code = NotFound desc = could not find container \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": container with ID starting with 9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb not found: ID does not exist"
Sep 17 15:42:06.423642 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.423610    2581 scope.go:117] "RemoveContainer" containerID="38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"
Sep 17 15:42:06.423807 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:42:06.423792    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": container with ID starting with 38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff not found: ID does not exist" containerID="38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"
Sep 17 15:42:06.423842 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.423809    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"} err="failed to get container status \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": rpc error: code = NotFound desc = could not find container \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": container with ID starting with 38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff not found: ID does not exist"
Sep 17 15:42:06.423842 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.423821    2581 scope.go:117] "RemoveContainer" containerID="b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"
Sep 17 15:42:06.424020 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:42:06.424002    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": container with ID starting with b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c not found: ID does not exist" containerID="b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"
Sep 17 15:42:06.424057 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424026    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"} err="failed to get container status \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": rpc error: code = NotFound desc = could not find container \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": container with ID starting with b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c not found: ID does not exist"
Sep 17 15:42:06.424057 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424043    2581 scope.go:117] "RemoveContainer" containerID="d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"
Sep 17 15:42:06.424276 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:42:06.424260    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": container with ID starting with d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433 not found: ID does not exist" containerID="d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"
Sep 17 15:42:06.424317 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424279    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"} err="failed to get container status \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": rpc error: code = NotFound desc = could not find container \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": container with ID starting with d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433 not found: ID does not exist"
Sep 17 15:42:06.424317 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424291    2581 scope.go:117] "RemoveContainer" containerID="46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"
Sep 17 15:42:06.424489 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:42:06.424473    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": container with ID starting with 46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb not found: ID does not exist" containerID="46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"
Sep 17 15:42:06.424526 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424492    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"} err="failed to get container status \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": rpc error: code = NotFound desc = could not find container \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": container with ID starting with 46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb not found: ID does not exist"
Sep 17 15:42:06.424526 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424503    2581 scope.go:117] "RemoveContainer" containerID="e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"
Sep 17 15:42:06.424715 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424697    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"} err="failed to get container status \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": rpc error: code = NotFound desc = could not find container \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": container with ID starting with e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764 not found: ID does not exist"
Sep 17 15:42:06.424767 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424715    2581 scope.go:117] "RemoveContainer" containerID="574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"
Sep 17 15:42:06.424975 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424958    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"} err="failed to get container status \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": rpc error: code = NotFound desc = could not find container \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": container with ID starting with 574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6 not found: ID does not exist"
Sep 17 15:42:06.425017 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.424976    2581 scope.go:117] "RemoveContainer" containerID="9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"
Sep 17 15:42:06.425206 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.425188    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"} err="failed to get container status \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": rpc error: code = NotFound desc = could not find container \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": container with ID starting with 9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb not found: ID does not exist"
Sep 17 15:42:06.425273 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.425206    2581 scope.go:117] "RemoveContainer" containerID="38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"
Sep 17 15:42:06.425414 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.425395    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"} err="failed to get container status \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": rpc error: code = NotFound desc = could not find container \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": container with ID starting with 38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff not found: ID does not exist"
Sep 17 15:42:06.425498 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.425416    2581 scope.go:117] "RemoveContainer" containerID="b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"
Sep 17 15:42:06.425679 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.425663    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"} err="failed to get container status \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": rpc error: code = NotFound desc = could not find container \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": container with ID starting with b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c not found: ID does not exist"
Sep 17 15:42:06.425679 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.425677    2581 scope.go:117] "RemoveContainer" containerID="d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"
Sep 17 15:42:06.425896 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.425878    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"} err="failed to get container status \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": rpc error: code = NotFound desc = could not find container \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": container with ID starting with d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433 not found: ID does not exist"
Sep 17 15:42:06.425936 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.425897    2581 scope.go:117] "RemoveContainer" containerID="46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"
Sep 17 15:42:06.426122 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426104    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"} err="failed to get container status \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": rpc error: code = NotFound desc = could not find container \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": container with ID starting with 46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb not found: ID does not exist"
Sep 17 15:42:06.426162 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426123    2581 scope.go:117] "RemoveContainer" containerID="e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"
Sep 17 15:42:06.426324 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426303    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"} err="failed to get container status \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": rpc error: code = NotFound desc = could not find container \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": container with ID starting with e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764 not found: ID does not exist"
Sep 17 15:42:06.426363 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426325    2581 scope.go:117] "RemoveContainer" containerID="574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"
Sep 17 15:42:06.426574 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426558    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"} err="failed to get container status \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": rpc error: code = NotFound desc = could not find container \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": container with ID starting with 574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6 not found: ID does not exist"
Sep 17 15:42:06.426643 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426574    2581 scope.go:117] "RemoveContainer" containerID="9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"
Sep 17 15:42:06.426748 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426733    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"} err="failed to get container status \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": rpc error: code = NotFound desc = could not find container \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": container with ID starting with 9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb not found: ID does not exist"
Sep 17 15:42:06.426801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426748    2581 scope.go:117] "RemoveContainer" containerID="38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"
Sep 17 15:42:06.426921 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426905    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"} err="failed to get container status \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": rpc error: code = NotFound desc = could not find container \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": container with ID starting with 38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff not found: ID does not exist"
Sep 17 15:42:06.426965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.426922    2581 scope.go:117] "RemoveContainer" containerID="b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"
Sep 17 15:42:06.427138 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427122    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"} err="failed to get container status \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": rpc error: code = NotFound desc = could not find container \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": container with ID starting with b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c not found: ID does not exist"
Sep 17 15:42:06.427189 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427138    2581 scope.go:117] "RemoveContainer" containerID="d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"
Sep 17 15:42:06.427363 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427343    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"} err="failed to get container status \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": rpc error: code = NotFound desc = could not find container \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": container with ID starting with d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433 not found: ID does not exist"
Sep 17 15:42:06.427402 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427364    2581 scope.go:117] "RemoveContainer" containerID="46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"
Sep 17 15:42:06.427573 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427558    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"} err="failed to get container status \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": rpc error: code = NotFound desc = could not find container \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": container with ID starting with 46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb not found: ID does not exist"
Sep 17 15:42:06.427615 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427573    2581 scope.go:117] "RemoveContainer" containerID="e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"
Sep 17 15:42:06.427756 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427740    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"} err="failed to get container status \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": rpc error: code = NotFound desc = could not find container \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": container with ID starting with e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764 not found: ID does not exist"
Sep 17 15:42:06.427805 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427755    2581 scope.go:117] "RemoveContainer" containerID="574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"
Sep 17 15:42:06.427945 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427924    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"} err="failed to get container status \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": rpc error: code = NotFound desc = could not find container \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": container with ID starting with 574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6 not found: ID does not exist"
Sep 17 15:42:06.427987 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.427946    2581 scope.go:117] "RemoveContainer" containerID="9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"
Sep 17 15:42:06.428132 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428112    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"} err="failed to get container status \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": rpc error: code = NotFound desc = could not find container \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": container with ID starting with 9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb not found: ID does not exist"
Sep 17 15:42:06.428174 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428133    2581 scope.go:117] "RemoveContainer" containerID="38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"
Sep 17 15:42:06.428319 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428305    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"} err="failed to get container status \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": rpc error: code = NotFound desc = could not find container \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": container with ID starting with 38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff not found: ID does not exist"
Sep 17 15:42:06.428403 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428319    2581 scope.go:117] "RemoveContainer" containerID="b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"
Sep 17 15:42:06.428516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428502    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"} err="failed to get container status \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": rpc error: code = NotFound desc = could not find container \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": container with ID starting with b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c not found: ID does not exist"
Sep 17 15:42:06.428516 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428515    2581 scope.go:117] "RemoveContainer" containerID="d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"
Sep 17 15:42:06.428717 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428682    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"} err="failed to get container status \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": rpc error: code = NotFound desc = could not find container \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": container with ID starting with d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433 not found: ID does not exist"
Sep 17 15:42:06.428717 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428701    2581 scope.go:117] "RemoveContainer" containerID="46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"
Sep 17 15:42:06.428946 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428929    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"} err="failed to get container status \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": rpc error: code = NotFound desc = could not find container \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": container with ID starting with 46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb not found: ID does not exist"
Sep 17 15:42:06.428999 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.428947    2581 scope.go:117] "RemoveContainer" containerID="e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"
Sep 17 15:42:06.429256 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.429191    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"} err="failed to get container status \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": rpc error: code = NotFound desc = could not find container \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": container with ID starting with e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764 not found: ID does not exist"
Sep 17 15:42:06.429256 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.429220    2581 scope.go:117] "RemoveContainer" containerID="574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"
Sep 17 15:42:06.429592 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.429568    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"} err="failed to get container status \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": rpc error: code = NotFound desc = could not find container \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": container with ID starting with 574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6 not found: ID does not exist"
Sep 17 15:42:06.429592 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.429592    2581 scope.go:117] "RemoveContainer" containerID="9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"
Sep 17 15:42:06.429971 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.429946    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"} err="failed to get container status \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": rpc error: code = NotFound desc = could not find container \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": container with ID starting with 9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb not found: ID does not exist"
Sep 17 15:42:06.429971 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.429971    2581 scope.go:117] "RemoveContainer" containerID="38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"
Sep 17 15:42:06.430212 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.430195    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"} err="failed to get container status \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": rpc error: code = NotFound desc = could not find container \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": container with ID starting with 38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff not found: ID does not exist"
Sep 17 15:42:06.430251 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.430212    2581 scope.go:117] "RemoveContainer" containerID="b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"
Sep 17 15:42:06.430458 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.430434    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"} err="failed to get container status \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": rpc error: code = NotFound desc = could not find container \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": container with ID starting with b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c not found: ID does not exist"
Sep 17 15:42:06.430544 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.430484    2581 scope.go:117] "RemoveContainer" containerID="d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"
Sep 17 15:42:06.430737 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.430717    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"} err="failed to get container status \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": rpc error: code = NotFound desc = could not find container \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": container with ID starting with d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433 not found: ID does not exist"
Sep 17 15:42:06.430798 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.430736    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:06.430843 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.430739    2581 scope.go:117] "RemoveContainer" containerID="46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"
Sep 17 15:42:06.431024 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431012    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="prometheus"
Sep 17 15:42:06.431070 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431028    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="prometheus"
Sep 17 15:42:06.431070 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431044    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy-web"
Sep 17 15:42:06.431070 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431051    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy-web"
Sep 17 15:42:06.431070 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431049    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"} err="failed to get container status \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": rpc error: code = NotFound desc = could not find container \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": container with ID starting with 46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb not found: ID does not exist"
Sep 17 15:42:06.431070 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431063    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="config-reloader"
Sep 17 15:42:06.431070 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431064    2581 scope.go:117] "RemoveContainer" containerID="e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"
Sep 17 15:42:06.431070 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431069    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="config-reloader"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431078    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy-thanos"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431085    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy-thanos"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431094    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="init-config-reloader"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431099    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="init-config-reloader"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431105    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="thanos-sidecar"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431110    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="thanos-sidecar"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431116    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431120    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431177    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy-web"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431185    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="thanos-sidecar"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431191    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431197    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="prometheus"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431203    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="config-reloader"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431209    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" containerName="kube-rbac-proxy-thanos"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431273    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764"} err="failed to get container status \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": rpc error: code = NotFound desc = could not find container \"e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764\": container with ID starting with e9898867691f6e0768bdd440d0e4ebcfcf4a1384bbb1ad589ddf4358e585b764 not found: ID does not exist"
Sep 17 15:42:06.431395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431289    2581 scope.go:117] "RemoveContainer" containerID="574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"
Sep 17 15:42:06.432025 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431518    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6"} err="failed to get container status \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": rpc error: code = NotFound desc = could not find container \"574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6\": container with ID starting with 574604bdb502c08aaf034f5b5425340c0514c7ff989e27c143da58e462493fe6 not found: ID does not exist"
Sep 17 15:42:06.432025 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431533    2581 scope.go:117] "RemoveContainer" containerID="9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"
Sep 17 15:42:06.432025 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431763    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb"} err="failed to get container status \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": rpc error: code = NotFound desc = could not find container \"9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb\": container with ID starting with 9190654755bc0e8f690e8eabdff39472de5cafcd2bcc3e8be1cf2424e217d1eb not found: ID does not exist"
Sep 17 15:42:06.432025 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.431789    2581 scope.go:117] "RemoveContainer" containerID="38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"
Sep 17 15:42:06.432152 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.432025    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff"} err="failed to get container status \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": rpc error: code = NotFound desc = could not find container \"38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff\": container with ID starting with 38a677959de17d8987c4e7117a611dbe95b75a86333d9556bfda09b553f36dff not found: ID does not exist"
Sep 17 15:42:06.432152 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.432040    2581 scope.go:117] "RemoveContainer" containerID="b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"
Sep 17 15:42:06.432265 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.432250    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c"} err="failed to get container status \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": rpc error: code = NotFound desc = could not find container \"b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c\": container with ID starting with b2281f2ff1b33272cdd585e7d4374e6a5a6a12ff2ed7b687d9048ccff059710c not found: ID does not exist"
Sep 17 15:42:06.432301 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.432265    2581 scope.go:117] "RemoveContainer" containerID="d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"
Sep 17 15:42:06.432534 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.432519    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433"} err="failed to get container status \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": rpc error: code = NotFound desc = could not find container \"d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433\": container with ID starting with d8fce18d7f9ccc92dfd8a46308861df198cbc0ab45eb073e0777b0692f423433 not found: ID does not exist"
Sep 17 15:42:06.432582 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.432534    2581 scope.go:117] "RemoveContainer" containerID="46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"
Sep 17 15:42:06.432756 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.432738    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb"} err="failed to get container status \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": rpc error: code = NotFound desc = could not find container \"46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb\": container with ID starting with 46e108ffd67e2f1c15c07ff74771191745841932e473b85374d610dc72bcb5bb not found: ID does not exist"
Sep 17 15:42:06.436517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.436503    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.438258 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.438239    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pr68z\""
Sep 17 15:42:06.438341 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.438276    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 15:42:06.438407 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.438275    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 15:42:06.438644 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.438627    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 15:42:06.438706 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.438629    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 15:42:06.438753 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.438632    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 15:42:06.439231 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.439215    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 15:42:06.439410 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.439387    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 15:42:06.439540 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.439405    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 15:42:06.439540 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.439404    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 15:42:06.439540 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.439405    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 15:42:06.440046 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.440026    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 15:42:06.440309 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.440291    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 15:42:06.440397 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.440363    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 15:42:06.441543 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.441527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bhec0mvvuq16b\""
Sep 17 15:42:06.442439 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.442422    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 15:42:06.444829 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.444812    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 15:42:06.454607 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.454562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:06.514169 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514357 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514357 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514357 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514357 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514357 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe81b2e4-16f1-4338-b968-c9fcc694c207-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-config\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514456    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514538    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkl8\" (UniqueName: \"kubernetes.io/projected/fe81b2e4-16f1-4338-b968-c9fcc694c207-kube-api-access-vqkl8\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514677 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514630    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe81b2e4-16f1-4338-b968-c9fcc694c207-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514956 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514679    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514956 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514956 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514956 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514826    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.514956 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.514909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615518 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615643 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615541    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615643 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615577    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615643 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615643 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe81b2e4-16f1-4338-b968-c9fcc694c207-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-config\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615790    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.615846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616168 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkl8\" (UniqueName: \"kubernetes.io/projected/fe81b2e4-16f1-4338-b968-c9fcc694c207-kube-api-access-vqkl8\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616168 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615885    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616168 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616168 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe81b2e4-16f1-4338-b968-c9fcc694c207-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616168 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615951    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616405 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.616222    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616405 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.615960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616405 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.616291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616405 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.616318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616405 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.616326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616405 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.616346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616737 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.616491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.616737 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.616559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.617342 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.617313    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.618949 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.618921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-config\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.619600 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.619085    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.619600 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.619430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-web-config\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.619600 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.619434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.619600 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.619434    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fe81b2e4-16f1-4338-b968-c9fcc694c207-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.619600 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.619554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.620362 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.619907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.620362 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.619934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.620362 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.620212    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.621026 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.621008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fe81b2e4-16f1-4338-b968-c9fcc694c207-config-out\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.621595 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.621573    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.622788 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.622759    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/fe81b2e4-16f1-4338-b968-c9fcc694c207-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.623118 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.623099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/fe81b2e4-16f1-4338-b968-c9fcc694c207-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.624563 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.624546    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkl8\" (UniqueName: \"kubernetes.io/projected/fe81b2e4-16f1-4338-b968-c9fcc694c207-kube-api-access-vqkl8\") pod \"prometheus-k8s-0\" (UID: \"fe81b2e4-16f1-4338-b968-c9fcc694c207\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:06.746143 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:06.746056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:42:07.024316 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:07.024243    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 15:42:07.026319 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:42:07.026295    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe81b2e4_16f1_4338_b968_c9fcc694c207.slice/crio-d6b623239b006e719b907400543fccb51e3b0dec0e7838960016ebb426c7c8a1 WatchSource:0}: Error finding container d6b623239b006e719b907400543fccb51e3b0dec0e7838960016ebb426c7c8a1: Status 404 returned error can't find the container with id d6b623239b006e719b907400543fccb51e3b0dec0e7838960016ebb426c7c8a1
Sep 17 15:42:07.380851 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:07.380820    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe81b2e4-16f1-4338-b968-c9fcc694c207" containerID="b4ee514abb50c87513d18231f84c5068e6a432083f2345f2d6cf30d7a114bf77" exitCode=0
Sep 17 15:42:07.381195 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:07.380887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe81b2e4-16f1-4338-b968-c9fcc694c207","Type":"ContainerDied","Data":"b4ee514abb50c87513d18231f84c5068e6a432083f2345f2d6cf30d7a114bf77"}
Sep 17 15:42:07.381195 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:07.380936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe81b2e4-16f1-4338-b968-c9fcc694c207","Type":"ContainerStarted","Data":"d6b623239b006e719b907400543fccb51e3b0dec0e7838960016ebb426c7c8a1"}
Sep 17 15:42:07.559933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:07.559895    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f0cd02-3d74-41ca-8efc-fb55017af6c4" path="/var/lib/kubelet/pods/d5f0cd02-3d74-41ca-8efc-fb55017af6c4/volumes"
Sep 17 15:42:08.393033 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:08.392995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe81b2e4-16f1-4338-b968-c9fcc694c207","Type":"ContainerStarted","Data":"9fe93f0355d09599a1704ba6d3f943daaeeb3db6530a4c1ed3be2d6ff6bc8cb9"}
Sep 17 15:42:08.393505 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:08.393105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe81b2e4-16f1-4338-b968-c9fcc694c207","Type":"ContainerStarted","Data":"67a905bd8ae60b490bbb2375564896ac1aeba26802fb1e129c368eb6d51166b6"}
Sep 17 15:42:08.393505 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:08.393149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe81b2e4-16f1-4338-b968-c9fcc694c207","Type":"ContainerStarted","Data":"b0dfc74d98124ea89d54c6903a6c066a1089952b6488187df8ac555b4cba2b6a"}
Sep 17 15:42:08.393505 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:08.393200    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe81b2e4-16f1-4338-b968-c9fcc694c207","Type":"ContainerStarted","Data":"5ef02863b2ebad1e2264223df5b774e73ca9c1b7e3c7691eece15dd4483da1f3"}
Sep 17 15:42:08.393505 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:08.393236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe81b2e4-16f1-4338-b968-c9fcc694c207","Type":"ContainerStarted","Data":"c3fc205269d8f1b0f3d902a2069ceb88150f6e46e4bb6db826977de010f88f27"}
Sep 17 15:42:08.393505 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:08.393274    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"fe81b2e4-16f1-4338-b968-c9fcc694c207","Type":"ContainerStarted","Data":"7869bbc5aed45f5c78ae5788cf2cb3589e6ae127c9f10f57b2885af72fdf3a4c"}
Sep 17 15:42:08.419643 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:08.419588    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.398737029 podStartE2EDuration="2.419570874s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.833845ms" isStatefulPod=false podCreationTimestamp="2026-09-17 15:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:42:08.417705612 +0000 UTC m=+219.341280179" watchObservedRunningTime="2026-09-17 15:42:08.419570874 +0000 UTC m=+219.343145440"
Sep 17 15:42:11.747134 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:42:11.747095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:43:06.747050 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:43:06.747019    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:43:06.762084 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:43:06.762061    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:43:07.641448 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:43:07.641419    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 15:43:29.513822 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:43:29.513792    2581 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 15:48:20.811433 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.811401    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-vj6dr"]
Sep 17 15:48:20.814670 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.814650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vj6dr"
Sep 17 15:48:20.817603 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.817583    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 15:48:20.817717 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.817646    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dl92p\""
Sep 17 15:48:20.817717 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.817583    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 15:48:20.817717 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.817705    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 15:48:20.817876 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.817759    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 15:48:20.824324 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.824302    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vj6dr"]
Sep 17 15:48:20.911330 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.911298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltdnm\" (UniqueName: \"kubernetes.io/projected/eb4bc29a-b99e-4107-964a-916507799d0c-kube-api-access-ltdnm\") pod \"s3-init-vj6dr\" (UID: \"eb4bc29a-b99e-4107-964a-916507799d0c\") " pod="kserve/s3-init-vj6dr"
Sep 17 15:48:20.911330 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:20.911332    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4bc29a-b99e-4107-964a-916507799d0c-model-cache\") pod \"s3-init-vj6dr\" (UID: \"eb4bc29a-b99e-4107-964a-916507799d0c\") " pod="kserve/s3-init-vj6dr"
Sep 17 15:48:21.012670 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:21.012637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltdnm\" (UniqueName: \"kubernetes.io/projected/eb4bc29a-b99e-4107-964a-916507799d0c-kube-api-access-ltdnm\") pod \"s3-init-vj6dr\" (UID: \"eb4bc29a-b99e-4107-964a-916507799d0c\") " pod="kserve/s3-init-vj6dr"
Sep 17 15:48:21.012670 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:21.012674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4bc29a-b99e-4107-964a-916507799d0c-model-cache\") pod \"s3-init-vj6dr\" (UID: \"eb4bc29a-b99e-4107-964a-916507799d0c\") " pod="kserve/s3-init-vj6dr"
Sep 17 15:48:21.013081 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:21.013063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4bc29a-b99e-4107-964a-916507799d0c-model-cache\") pod \"s3-init-vj6dr\" (UID: \"eb4bc29a-b99e-4107-964a-916507799d0c\") " pod="kserve/s3-init-vj6dr"
Sep 17 15:48:21.021323 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:21.021298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltdnm\" (UniqueName: \"kubernetes.io/projected/eb4bc29a-b99e-4107-964a-916507799d0c-kube-api-access-ltdnm\") pod \"s3-init-vj6dr\" (UID: \"eb4bc29a-b99e-4107-964a-916507799d0c\") " pod="kserve/s3-init-vj6dr"
Sep 17 15:48:21.135485 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:21.135383    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vj6dr"
Sep 17 15:48:21.408766 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:21.408689    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-vj6dr"]
Sep 17 15:48:21.413229 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:48:21.413197    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeb4bc29a_b99e_4107_964a_916507799d0c.slice/crio-63cdab40ad7320c3917b9fe444dcfb4969b763d6af78cef747646a63bf91d931 WatchSource:0}: Error finding container 63cdab40ad7320c3917b9fe444dcfb4969b763d6af78cef747646a63bf91d931: Status 404 returned error can't find the container with id 63cdab40ad7320c3917b9fe444dcfb4969b763d6af78cef747646a63bf91d931
Sep 17 15:48:21.415180 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:21.415161    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:48:21.811981 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:21.811941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vj6dr" event={"ID":"eb4bc29a-b99e-4107-964a-916507799d0c","Type":"ContainerStarted","Data":"63cdab40ad7320c3917b9fe444dcfb4969b763d6af78cef747646a63bf91d931"}
Sep 17 15:48:25.832210 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:25.832175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vj6dr" event={"ID":"eb4bc29a-b99e-4107-964a-916507799d0c","Type":"ContainerStarted","Data":"fc330f72ba02a0e970594592465e6d795a70066080b8e0fa69e3c798db854160"}
Sep 17 15:48:27.841852 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:27.841818    2581 generic.go:356] "Generic (PLEG): container finished" podID="eb4bc29a-b99e-4107-964a-916507799d0c" containerID="fc330f72ba02a0e970594592465e6d795a70066080b8e0fa69e3c798db854160" exitCode=0
Sep 17 15:48:27.842314 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:27.841896    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vj6dr" event={"ID":"eb4bc29a-b99e-4107-964a-916507799d0c","Type":"ContainerDied","Data":"fc330f72ba02a0e970594592465e6d795a70066080b8e0fa69e3c798db854160"}
Sep 17 15:48:32.247520 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:32.247496    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 17 15:48:32.868199 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:32.868164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vj6dr" event={"ID":"eb4bc29a-b99e-4107-964a-916507799d0c","Type":"ContainerStarted","Data":"fae3a66b12bd279d9a44e5f9d10db78e44de396864974c63bcc10d0ea3a74ffb"}
Sep 17 15:48:32.888059 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:48:32.888014    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-vj6dr" podStartSLOduration=3.026489833 podStartE2EDuration="12.887997566s" totalImagesPullingTime="7.829667629s" totalInitContainerRuntime="2.031840104s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:48:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:48:32.886110505 +0000 UTC m=+603.809685072" watchObservedRunningTime="2026-09-17 15:48:32.887997566 +0000 UTC m=+603.811572133"
Sep 17 15:49:38.120563 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:38.120529    2581 generic.go:356] "Generic (PLEG): container finished" podID="eb4bc29a-b99e-4107-964a-916507799d0c" containerID="fae3a66b12bd279d9a44e5f9d10db78e44de396864974c63bcc10d0ea3a74ffb" exitCode=0
Sep 17 15:49:38.121001 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:38.120601    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vj6dr" event={"ID":"eb4bc29a-b99e-4107-964a-916507799d0c","Type":"ContainerDied","Data":"fae3a66b12bd279d9a44e5f9d10db78e44de396864974c63bcc10d0ea3a74ffb"}
Sep 17 15:49:39.263157 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:39.263133    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vj6dr"
Sep 17 15:49:39.360529 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:39.360499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4bc29a-b99e-4107-964a-916507799d0c-model-cache\") pod \"eb4bc29a-b99e-4107-964a-916507799d0c\" (UID: \"eb4bc29a-b99e-4107-964a-916507799d0c\") "
Sep 17 15:49:39.360676 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:39.360582    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltdnm\" (UniqueName: \"kubernetes.io/projected/eb4bc29a-b99e-4107-964a-916507799d0c-kube-api-access-ltdnm\") pod \"eb4bc29a-b99e-4107-964a-916507799d0c\" (UID: \"eb4bc29a-b99e-4107-964a-916507799d0c\") "
Sep 17 15:49:39.362791 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:39.362760    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4bc29a-b99e-4107-964a-916507799d0c-kube-api-access-ltdnm" (OuterVolumeSpecName: "kube-api-access-ltdnm") pod "eb4bc29a-b99e-4107-964a-916507799d0c" (UID: "eb4bc29a-b99e-4107-964a-916507799d0c"). InnerVolumeSpecName "kube-api-access-ltdnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:39.379937 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:39.379886    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb4bc29a-b99e-4107-964a-916507799d0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "eb4bc29a-b99e-4107-964a-916507799d0c" (UID: "eb4bc29a-b99e-4107-964a-916507799d0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:39.461870 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:39.461847    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eb4bc29a-b99e-4107-964a-916507799d0c-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:49:39.461870 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:39.461868    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltdnm\" (UniqueName: \"kubernetes.io/projected/eb4bc29a-b99e-4107-964a-916507799d0c-kube-api-access-ltdnm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:49:40.130177 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:40.130141    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-vj6dr" event={"ID":"eb4bc29a-b99e-4107-964a-916507799d0c","Type":"ContainerDied","Data":"63cdab40ad7320c3917b9fe444dcfb4969b763d6af78cef747646a63bf91d931"}
Sep 17 15:49:40.130177 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:40.130178    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63cdab40ad7320c3917b9fe444dcfb4969b763d6af78cef747646a63bf91d931"
Sep 17 15:49:40.130177 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:40.130178    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-vj6dr"
Sep 17 15:49:47.623012 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.622978    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-rmkx2"]
Sep 17 15:49:47.623376 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.623280    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4bc29a-b99e-4107-964a-916507799d0c" containerName="s3-init"
Sep 17 15:49:47.623376 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.623291    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4bc29a-b99e-4107-964a-916507799d0c" containerName="s3-init"
Sep 17 15:49:47.623376 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.623307    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4bc29a-b99e-4107-964a-916507799d0c" containerName="download-hf-model"
Sep 17 15:49:47.623376 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.623313    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4bc29a-b99e-4107-964a-916507799d0c" containerName="download-hf-model"
Sep 17 15:49:47.623376 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.623362    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4bc29a-b99e-4107-964a-916507799d0c" containerName="s3-init"
Sep 17 15:49:47.626583 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.626567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:47.628477 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.628432    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 15:49:47.628602 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.628547    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 15:49:47.628656 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.628651    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 15:49:47.628743 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.628728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dl92p\""
Sep 17 15:49:47.628916 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.628903    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 15:49:47.634785 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.634760    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-rmkx2"]
Sep 17 15:49:47.729690 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.729664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-model-cache\") pod \"s3-tls-init-custom-rmkx2\" (UID: \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\") " pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:47.729819 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.729701    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-885v4\" (UniqueName: \"kubernetes.io/projected/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-kube-api-access-885v4\") pod \"s3-tls-init-custom-rmkx2\" (UID: \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\") " pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:47.830093 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.830057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-885v4\" (UniqueName: \"kubernetes.io/projected/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-kube-api-access-885v4\") pod \"s3-tls-init-custom-rmkx2\" (UID: \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\") " pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:47.830250 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.830133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-model-cache\") pod \"s3-tls-init-custom-rmkx2\" (UID: \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\") " pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:47.830436 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.830419    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-model-cache\") pod \"s3-tls-init-custom-rmkx2\" (UID: \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\") " pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:47.838742 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.838716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-885v4\" (UniqueName: \"kubernetes.io/projected/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-kube-api-access-885v4\") pod \"s3-tls-init-custom-rmkx2\" (UID: \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\") " pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:47.945007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:47.944949    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:48.217943 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:48.217917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-rmkx2"]
Sep 17 15:49:48.220565 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:49:48.220535    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8611437_ad9d_4aa3_bb8d_6bab1452ac35.slice/crio-d3d106373125bba4e135bcd67eae0fec414265de698820473b040972c45571fc WatchSource:0}: Error finding container d3d106373125bba4e135bcd67eae0fec414265de698820473b040972c45571fc: Status 404 returned error can't find the container with id d3d106373125bba4e135bcd67eae0fec414265de698820473b040972c45571fc
Sep 17 15:49:49.167949 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:49.167888    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmkx2" event={"ID":"f8611437-ad9d-4aa3-bb8d-6bab1452ac35","Type":"ContainerStarted","Data":"f2e06a94159ce98fea17ee9ba62163c8cc1136aa086225efb375c3dfa319ec02"}
Sep 17 15:49:49.167949 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:49.167930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmkx2" event={"ID":"f8611437-ad9d-4aa3-bb8d-6bab1452ac35","Type":"ContainerStarted","Data":"d3d106373125bba4e135bcd67eae0fec414265de698820473b040972c45571fc"}
Sep 17 15:49:50.173611 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:50.173584    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8611437-ad9d-4aa3-bb8d-6bab1452ac35" containerID="f2e06a94159ce98fea17ee9ba62163c8cc1136aa086225efb375c3dfa319ec02" exitCode=0
Sep 17 15:49:50.173922 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:50.173674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmkx2" event={"ID":"f8611437-ad9d-4aa3-bb8d-6bab1452ac35","Type":"ContainerDied","Data":"f2e06a94159ce98fea17ee9ba62163c8cc1136aa086225efb375c3dfa319ec02"}
Sep 17 15:49:51.179971 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:51.179941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmkx2" event={"ID":"f8611437-ad9d-4aa3-bb8d-6bab1452ac35","Type":"ContainerStarted","Data":"6edecbd3da618fa2c1fa16b3f442bf658cb918e98a2df712917990af21558a5e"}
Sep 17 15:49:51.195932 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:51.195888    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-rmkx2" podStartSLOduration=2.351076274 podStartE2EDuration="4.195872248s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.844795974s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:49:51.195276608 +0000 UTC m=+682.118851173" watchObservedRunningTime="2026-09-17 15:49:51.195872248 +0000 UTC m=+682.119446821"
Sep 17 15:49:56.203786 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:56.203751    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8611437-ad9d-4aa3-bb8d-6bab1452ac35" containerID="6edecbd3da618fa2c1fa16b3f442bf658cb918e98a2df712917990af21558a5e" exitCode=0
Sep 17 15:49:56.204136 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:56.203820    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmkx2" event={"ID":"f8611437-ad9d-4aa3-bb8d-6bab1452ac35","Type":"ContainerDied","Data":"6edecbd3da618fa2c1fa16b3f442bf658cb918e98a2df712917990af21558a5e"}
Sep 17 15:49:57.336946 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:57.336926    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:57.415800 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:57.415772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-885v4\" (UniqueName: \"kubernetes.io/projected/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-kube-api-access-885v4\") pod \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\" (UID: \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\") "
Sep 17 15:49:57.415962 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:57.415836    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-model-cache\") pod \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\" (UID: \"f8611437-ad9d-4aa3-bb8d-6bab1452ac35\") "
Sep 17 15:49:57.417883 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:57.417859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-kube-api-access-885v4" (OuterVolumeSpecName: "kube-api-access-885v4") pod "f8611437-ad9d-4aa3-bb8d-6bab1452ac35" (UID: "f8611437-ad9d-4aa3-bb8d-6bab1452ac35"). InnerVolumeSpecName "kube-api-access-885v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:49:57.438206 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:57.438180    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-model-cache" (OuterVolumeSpecName: "model-cache") pod "f8611437-ad9d-4aa3-bb8d-6bab1452ac35" (UID: "f8611437-ad9d-4aa3-bb8d-6bab1452ac35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:49:57.517130 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:57.517062    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-885v4\" (UniqueName: \"kubernetes.io/projected/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-kube-api-access-885v4\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:49:57.517130 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:57.517087    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8611437-ad9d-4aa3-bb8d-6bab1452ac35-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:49:58.216673 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:58.216644    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-rmkx2"
Sep 17 15:49:58.216840 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:58.216677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-rmkx2" event={"ID":"f8611437-ad9d-4aa3-bb8d-6bab1452ac35","Type":"ContainerDied","Data":"d3d106373125bba4e135bcd67eae0fec414265de698820473b040972c45571fc"}
Sep 17 15:49:58.216840 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:49:58.216709    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3d106373125bba4e135bcd67eae0fec414265de698820473b040972c45571fc"
Sep 17 15:50:01.021393 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.021355    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-vrjqm"]
Sep 17 15:50:01.021770 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.021667    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8611437-ad9d-4aa3-bb8d-6bab1452ac35" containerName="download-hf-model"
Sep 17 15:50:01.021770 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.021678    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8611437-ad9d-4aa3-bb8d-6bab1452ac35" containerName="download-hf-model"
Sep 17 15:50:01.021770 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.021698    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8611437-ad9d-4aa3-bb8d-6bab1452ac35" containerName="s3-tls-init-custom"
Sep 17 15:50:01.021770 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.021704    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8611437-ad9d-4aa3-bb8d-6bab1452ac35" containerName="s3-tls-init-custom"
Sep 17 15:50:01.021770 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.021751    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8611437-ad9d-4aa3-bb8d-6bab1452ac35" containerName="s3-tls-init-custom"
Sep 17 15:50:01.024958 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.024943    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:01.026806 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.026779    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 17 15:50:01.026946 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.026788    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 15:50:01.026946 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.026789    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dl92p\""
Sep 17 15:50:01.026946 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.026841    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 15:50:01.027238 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.027220    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 15:50:01.031591 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.031566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-vrjqm"]
Sep 17 15:50:01.148967 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.148921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl48v\" (UniqueName: \"kubernetes.io/projected/672ade60-2979-4e61-aab5-ff3652445d8d-kube-api-access-tl48v\") pod \"s3-tls-init-serving-vrjqm\" (UID: \"672ade60-2979-4e61-aab5-ff3652445d8d\") " pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:01.149147 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.149048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672ade60-2979-4e61-aab5-ff3652445d8d-model-cache\") pod \"s3-tls-init-serving-vrjqm\" (UID: \"672ade60-2979-4e61-aab5-ff3652445d8d\") " pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:01.250043 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.249998    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672ade60-2979-4e61-aab5-ff3652445d8d-model-cache\") pod \"s3-tls-init-serving-vrjqm\" (UID: \"672ade60-2979-4e61-aab5-ff3652445d8d\") " pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:01.250208 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.250054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl48v\" (UniqueName: \"kubernetes.io/projected/672ade60-2979-4e61-aab5-ff3652445d8d-kube-api-access-tl48v\") pod \"s3-tls-init-serving-vrjqm\" (UID: \"672ade60-2979-4e61-aab5-ff3652445d8d\") " pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:01.250407 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.250388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672ade60-2979-4e61-aab5-ff3652445d8d-model-cache\") pod \"s3-tls-init-serving-vrjqm\" (UID: \"672ade60-2979-4e61-aab5-ff3652445d8d\") " pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:01.257762 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.257739    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl48v\" (UniqueName: \"kubernetes.io/projected/672ade60-2979-4e61-aab5-ff3652445d8d-kube-api-access-tl48v\") pod \"s3-tls-init-serving-vrjqm\" (UID: \"672ade60-2979-4e61-aab5-ff3652445d8d\") " pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:01.346314 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.346283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:01.614940 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:01.614825    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-vrjqm"]
Sep 17 15:50:01.617673 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:50:01.617643    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod672ade60_2979_4e61_aab5_ff3652445d8d.slice/crio-f8003704a0b4e45854c028c33591bffc7a451a4d1e1a28b3d358d1e1cb21b125 WatchSource:0}: Error finding container f8003704a0b4e45854c028c33591bffc7a451a4d1e1a28b3d358d1e1cb21b125: Status 404 returned error can't find the container with id f8003704a0b4e45854c028c33591bffc7a451a4d1e1a28b3d358d1e1cb21b125
Sep 17 15:50:02.235152 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:02.235113    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vrjqm" event={"ID":"672ade60-2979-4e61-aab5-ff3652445d8d","Type":"ContainerStarted","Data":"f400d09513760373d56af77f5019aa1eb744701fb1cdc173408520d2a73edc56"}
Sep 17 15:50:02.235547 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:02.235165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vrjqm" event={"ID":"672ade60-2979-4e61-aab5-ff3652445d8d","Type":"ContainerStarted","Data":"f8003704a0b4e45854c028c33591bffc7a451a4d1e1a28b3d358d1e1cb21b125"}
Sep 17 15:50:04.245201 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:04.245166    2581 generic.go:356] "Generic (PLEG): container finished" podID="672ade60-2979-4e61-aab5-ff3652445d8d" containerID="f400d09513760373d56af77f5019aa1eb744701fb1cdc173408520d2a73edc56" exitCode=0
Sep 17 15:50:04.245565 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:04.245239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vrjqm" event={"ID":"672ade60-2979-4e61-aab5-ff3652445d8d","Type":"ContainerDied","Data":"f400d09513760373d56af77f5019aa1eb744701fb1cdc173408520d2a73edc56"}
Sep 17 15:50:05.251645 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:05.251609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vrjqm" event={"ID":"672ade60-2979-4e61-aab5-ff3652445d8d","Type":"ContainerStarted","Data":"94e776a19b6c3f8d8e7999159bf4e84d16e5fd74ebcddaad43686c8f1512921a"}
Sep 17 15:50:05.270805 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:05.270759    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-vrjqm" podStartSLOduration=2.272493608 podStartE2EDuration="4.270744158s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.99825055s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:05.269794366 +0000 UTC m=+696.193368933" watchObservedRunningTime="2026-09-17 15:50:05.270744158 +0000 UTC m=+696.194318764"
Sep 17 15:50:11.280257 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:11.280222    2581 generic.go:356] "Generic (PLEG): container finished" podID="672ade60-2979-4e61-aab5-ff3652445d8d" containerID="94e776a19b6c3f8d8e7999159bf4e84d16e5fd74ebcddaad43686c8f1512921a" exitCode=0
Sep 17 15:50:11.280708 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:11.280293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vrjqm" event={"ID":"672ade60-2979-4e61-aab5-ff3652445d8d","Type":"ContainerDied","Data":"94e776a19b6c3f8d8e7999159bf4e84d16e5fd74ebcddaad43686c8f1512921a"}
Sep 17 15:50:12.413942 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:12.413918    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:12.436242 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:12.436220    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl48v\" (UniqueName: \"kubernetes.io/projected/672ade60-2979-4e61-aab5-ff3652445d8d-kube-api-access-tl48v\") pod \"672ade60-2979-4e61-aab5-ff3652445d8d\" (UID: \"672ade60-2979-4e61-aab5-ff3652445d8d\") "
Sep 17 15:50:12.436374 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:12.436281    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672ade60-2979-4e61-aab5-ff3652445d8d-model-cache\") pod \"672ade60-2979-4e61-aab5-ff3652445d8d\" (UID: \"672ade60-2979-4e61-aab5-ff3652445d8d\") "
Sep 17 15:50:12.438530 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:12.438501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672ade60-2979-4e61-aab5-ff3652445d8d-kube-api-access-tl48v" (OuterVolumeSpecName: "kube-api-access-tl48v") pod "672ade60-2979-4e61-aab5-ff3652445d8d" (UID: "672ade60-2979-4e61-aab5-ff3652445d8d"). InnerVolumeSpecName "kube-api-access-tl48v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:50:12.480161 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:12.480130    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/672ade60-2979-4e61-aab5-ff3652445d8d-model-cache" (OuterVolumeSpecName: "model-cache") pod "672ade60-2979-4e61-aab5-ff3652445d8d" (UID: "672ade60-2979-4e61-aab5-ff3652445d8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:50:12.537458 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:12.537435    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/672ade60-2979-4e61-aab5-ff3652445d8d-model-cache\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:50:12.537586 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:12.537477    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl48v\" (UniqueName: \"kubernetes.io/projected/672ade60-2979-4e61-aab5-ff3652445d8d-kube-api-access-tl48v\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:50:13.289614 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:13.289583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-vrjqm" event={"ID":"672ade60-2979-4e61-aab5-ff3652445d8d","Type":"ContainerDied","Data":"f8003704a0b4e45854c028c33591bffc7a451a4d1e1a28b3d358d1e1cb21b125"}
Sep 17 15:50:13.289614 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:13.289619    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8003704a0b4e45854c028c33591bffc7a451a4d1e1a28b3d358d1e1cb21b125"
Sep 17 15:50:13.289838 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:13.289622    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-vrjqm"
Sep 17 15:50:25.829709 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.829624    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"]
Sep 17 15:50:25.830234 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.830109    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672ade60-2979-4e61-aab5-ff3652445d8d" containerName="download-hf-model"
Sep 17 15:50:25.830234 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.830127    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="672ade60-2979-4e61-aab5-ff3652445d8d" containerName="download-hf-model"
Sep 17 15:50:25.830234 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.830142    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672ade60-2979-4e61-aab5-ff3652445d8d" containerName="s3-tls-init-serving"
Sep 17 15:50:25.830234 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.830150    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="672ade60-2979-4e61-aab5-ff3652445d8d" containerName="s3-tls-init-serving"
Sep 17 15:50:25.830234 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.830227    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="672ade60-2979-4e61-aab5-ff3652445d8d" containerName="s3-tls-init-serving"
Sep 17 15:50:25.833613 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.833592    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:25.835818 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.835797    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 17 15:50:25.836046 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.836025    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:50:25.836046 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.836033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 17 15:50:25.836183 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.836030    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 15:50:25.836183 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.836062    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 15:50:25.836183 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.836070    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 15:50:25.836552 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.836539    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 15:50:25.845506 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.845486    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"]
Sep 17 15:50:25.929706 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.929677    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"]
Sep 17 15:50:25.933213 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.933198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:25.935209 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.935188    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f9f2bc-predictor-serving-cert\""
Sep 17 15:50:25.935305 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.935191    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-f9f2bc\""
Sep 17 15:50:25.935305 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.935191    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-f9f2bc-dockercfg-tqsnt\""
Sep 17 15:50:25.935548 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.935535    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-f9f2bc-kube-rbac-proxy-sar-config\""
Sep 17 15:50:25.940877 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.940859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d923b54-fbde-4925-80b3-587dd361db88-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:25.940971 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.940884    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25d28\" (UniqueName: \"kubernetes.io/projected/0d923b54-fbde-4925-80b3-587dd361db88-kube-api-access-25d28\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:25.940971 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.940923    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:25.940971 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.940956    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d923b54-fbde-4925-80b3-587dd361db88-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:25.941118 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.941028    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:25.944477 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:25.944438    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"]
Sep 17 15:50:26.042107 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1058b8a-af45-425b-80f7-a442c43d20e9-kserve-provision-location\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.042251 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d923b54-fbde-4925-80b3-587dd361db88-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.042251 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.042251 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042174    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.042251 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042229    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1058b8a-af45-425b-80f7-a442c43d20e9-proxy-tls\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.042511 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d923b54-fbde-4925-80b3-587dd361db88-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.042511 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25d28\" (UniqueName: \"kubernetes.io/projected/0d923b54-fbde-4925-80b3-587dd361db88-kube-api-access-25d28\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.042511 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-cabundle-cert\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.042511 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:50:26.042381    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 17 15:50:26.042511 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.042511 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:50:26.042452    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d923b54-fbde-4925-80b3-587dd361db88-proxy-tls podName:0d923b54-fbde-4925-80b3-587dd361db88 nodeName:}" failed. No retries permitted until 2026-09-17 15:50:26.542431818 +0000 UTC m=+717.466006375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0d923b54-fbde-4925-80b3-587dd361db88-proxy-tls") pod "isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" (UID: "0d923b54-fbde-4925-80b3-587dd361db88") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Sep 17 15:50:26.042883 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042524    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm7d6\" (UniqueName: \"kubernetes.io/projected/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-api-access-hm7d6\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.042883 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.042838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.043061 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.043034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.043157 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.043088    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d923b54-fbde-4925-80b3-587dd361db88-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.052939 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.052921    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25d28\" (UniqueName: \"kubernetes.io/projected/0d923b54-fbde-4925-80b3-587dd361db88-kube-api-access-25d28\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.143163 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.143103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm7d6\" (UniqueName: \"kubernetes.io/projected/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-api-access-hm7d6\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.143163 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.143151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1058b8a-af45-425b-80f7-a442c43d20e9-kserve-provision-location\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.143318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.143179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.143318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.143213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1058b8a-af45-425b-80f7-a442c43d20e9-proxy-tls\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.143318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.143261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-cabundle-cert\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.143453 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:50:26.143373    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-serving-cert: secret "isvc-init-fail-f9f2bc-predictor-serving-cert" not found
Sep 17 15:50:26.143453 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:50:26.143432    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1058b8a-af45-425b-80f7-a442c43d20e9-proxy-tls podName:d1058b8a-af45-425b-80f7-a442c43d20e9 nodeName:}" failed. No retries permitted until 2026-09-17 15:50:26.643418771 +0000 UTC m=+717.566993316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1058b8a-af45-425b-80f7-a442c43d20e9-proxy-tls") pod "isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff" (UID: "d1058b8a-af45-425b-80f7-a442c43d20e9") : secret "isvc-init-fail-f9f2bc-predictor-serving-cert" not found
Sep 17 15:50:26.143568 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.143516    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1058b8a-af45-425b-80f7-a442c43d20e9-kserve-provision-location\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.143931 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.143911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-cabundle-cert\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.143978 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.143919    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.151786 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.151770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm7d6\" (UniqueName: \"kubernetes.io/projected/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-api-access-hm7d6\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.546728 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.546701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d923b54-fbde-4925-80b3-587dd361db88-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.549025 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.548996    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d923b54-fbde-4925-80b3-587dd361db88-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-6c74cc89d5-km88c\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.647737 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.647706    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1058b8a-af45-425b-80f7-a442c43d20e9-proxy-tls\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.650022 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.650001    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1058b8a-af45-425b-80f7-a442c43d20e9-proxy-tls\") pod \"isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") " pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:26.743499 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.743459    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:26.843276 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:26.843210    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:27.019106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:27.019075    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"]
Sep 17 15:50:27.022244 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:50:27.022221    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d923b54_fbde_4925_80b3_587dd361db88.slice/crio-46f6736ee72af2cb07e6777f58950e9fc690e2db96439df59d22e0723120d567 WatchSource:0}: Error finding container 46f6736ee72af2cb07e6777f58950e9fc690e2db96439df59d22e0723120d567: Status 404 returned error can't find the container with id 46f6736ee72af2cb07e6777f58950e9fc690e2db96439df59d22e0723120d567
Sep 17 15:50:27.116612 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:27.116543    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"]
Sep 17 15:50:27.120792 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:50:27.120756    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1058b8a_af45_425b_80f7_a442c43d20e9.slice/crio-db7cb961c848aa8a890fe8627f82943777db21228052c75ffe6f9a647424beca WatchSource:0}: Error finding container db7cb961c848aa8a890fe8627f82943777db21228052c75ffe6f9a647424beca: Status 404 returned error can't find the container with id db7cb961c848aa8a890fe8627f82943777db21228052c75ffe6f9a647424beca
Sep 17 15:50:27.354817 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:27.354785    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff" event={"ID":"d1058b8a-af45-425b-80f7-a442c43d20e9","Type":"ContainerStarted","Data":"5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81"}
Sep 17 15:50:27.355011 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:27.354855    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff" event={"ID":"d1058b8a-af45-425b-80f7-a442c43d20e9","Type":"ContainerStarted","Data":"db7cb961c848aa8a890fe8627f82943777db21228052c75ffe6f9a647424beca"}
Sep 17 15:50:27.356598 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:27.356575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerStarted","Data":"25ceb744b320043d47fc5f5ec13facc071686496373662e4b075ef5c7698b852"}
Sep 17 15:50:27.356709 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:27.356644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerStarted","Data":"46f6736ee72af2cb07e6777f58950e9fc690e2db96439df59d22e0723120d567"}
Sep 17 15:50:31.373977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:31.373905    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff_d1058b8a-af45-425b-80f7-a442c43d20e9/storage-initializer/0.log"
Sep 17 15:50:31.373977 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:31.373943    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1058b8a-af45-425b-80f7-a442c43d20e9" containerID="5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81" exitCode=1
Sep 17 15:50:31.374537 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:31.374001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff" event={"ID":"d1058b8a-af45-425b-80f7-a442c43d20e9","Type":"ContainerDied","Data":"5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81"}
Sep 17 15:50:31.375691 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:31.375669    2581 generic.go:356] "Generic (PLEG): container finished" podID="0d923b54-fbde-4925-80b3-587dd361db88" containerID="25ceb744b320043d47fc5f5ec13facc071686496373662e4b075ef5c7698b852" exitCode=0
Sep 17 15:50:31.375778 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:31.375745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerDied","Data":"25ceb744b320043d47fc5f5ec13facc071686496373662e4b075ef5c7698b852"}
Sep 17 15:50:32.387208 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:32.386308    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff_d1058b8a-af45-425b-80f7-a442c43d20e9/storage-initializer/0.log"
Sep 17 15:50:32.387208 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:32.386448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff" event={"ID":"d1058b8a-af45-425b-80f7-a442c43d20e9","Type":"ContainerStarted","Data":"f39959e75b56adaedb7f02736a92ae9f4cd36e41327aacaa9f467fffe850c6fc"}
Sep 17 15:50:35.573332 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:35.572724    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"]
Sep 17 15:50:35.573332 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:35.572968    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff" podUID="d1058b8a-af45-425b-80f7-a442c43d20e9" containerName="storage-initializer" containerID="cri-o://f39959e75b56adaedb7f02736a92ae9f4cd36e41327aacaa9f467fffe850c6fc" gracePeriod=30
Sep 17 15:50:36.414333 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:36.414308    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff_d1058b8a-af45-425b-80f7-a442c43d20e9/storage-initializer/1.log"
Sep 17 15:50:36.414782 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:36.414765    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff_d1058b8a-af45-425b-80f7-a442c43d20e9/storage-initializer/0.log"
Sep 17 15:50:36.414899 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:36.414801    2581 generic.go:356] "Generic (PLEG): container finished" podID="d1058b8a-af45-425b-80f7-a442c43d20e9" containerID="f39959e75b56adaedb7f02736a92ae9f4cd36e41327aacaa9f467fffe850c6fc" exitCode=1
Sep 17 15:50:36.414899 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:36.414873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff" event={"ID":"d1058b8a-af45-425b-80f7-a442c43d20e9","Type":"ContainerDied","Data":"f39959e75b56adaedb7f02736a92ae9f4cd36e41327aacaa9f467fffe850c6fc"}
Sep 17 15:50:36.415016 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:36.414944    2581 scope.go:117] "RemoveContainer" containerID="5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81"
Sep 17 15:50:39.856010 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.855985    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff_d1058b8a-af45-425b-80f7-a442c43d20e9/storage-initializer/1.log"
Sep 17 15:50:39.859004 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.858977    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:39.971336 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.971296    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-cabundle-cert\") pod \"d1058b8a-af45-425b-80f7-a442c43d20e9\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") "
Sep 17 15:50:39.971530 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.971344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-rbac-proxy-sar-config\") pod \"d1058b8a-af45-425b-80f7-a442c43d20e9\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") "
Sep 17 15:50:39.971530 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.971401    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1058b8a-af45-425b-80f7-a442c43d20e9-kserve-provision-location\") pod \"d1058b8a-af45-425b-80f7-a442c43d20e9\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") "
Sep 17 15:50:39.971530 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.971432    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm7d6\" (UniqueName: \"kubernetes.io/projected/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-api-access-hm7d6\") pod \"d1058b8a-af45-425b-80f7-a442c43d20e9\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") "
Sep 17 15:50:39.971530 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.971505    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1058b8a-af45-425b-80f7-a442c43d20e9-proxy-tls\") pod \"d1058b8a-af45-425b-80f7-a442c43d20e9\" (UID: \"d1058b8a-af45-425b-80f7-a442c43d20e9\") "
Sep 17 15:50:39.971768 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.971734    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1058b8a-af45-425b-80f7-a442c43d20e9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1058b8a-af45-425b-80f7-a442c43d20e9" (UID: "d1058b8a-af45-425b-80f7-a442c43d20e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:50:39.971815 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.971789    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1058b8a-af45-425b-80f7-a442c43d20e9" (UID: "d1058b8a-af45-425b-80f7-a442c43d20e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:50:39.971859 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.971808    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1058b8a-af45-425b-80f7-a442c43d20e9" (UID: "d1058b8a-af45-425b-80f7-a442c43d20e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:50:39.974375 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.974346    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1058b8a-af45-425b-80f7-a442c43d20e9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1058b8a-af45-425b-80f7-a442c43d20e9" (UID: "d1058b8a-af45-425b-80f7-a442c43d20e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:50:39.974554 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:39.974494    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-api-access-hm7d6" (OuterVolumeSpecName: "kube-api-access-hm7d6") pod "d1058b8a-af45-425b-80f7-a442c43d20e9" (UID: "d1058b8a-af45-425b-80f7-a442c43d20e9"). InnerVolumeSpecName "kube-api-access-hm7d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:50:40.072855 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.072820    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:50:40.072855 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.072852    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:50:40.072855 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.072862    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1058b8a-af45-425b-80f7-a442c43d20e9-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:50:40.073171 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.072870    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm7d6\" (UniqueName: \"kubernetes.io/projected/d1058b8a-af45-425b-80f7-a442c43d20e9-kube-api-access-hm7d6\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:50:40.073171 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.072880    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1058b8a-af45-425b-80f7-a442c43d20e9-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:50:40.435023 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.434995    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff_d1058b8a-af45-425b-80f7-a442c43d20e9/storage-initializer/1.log"
Sep 17 15:50:40.435513 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.435488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff" event={"ID":"d1058b8a-af45-425b-80f7-a442c43d20e9","Type":"ContainerDied","Data":"db7cb961c848aa8a890fe8627f82943777db21228052c75ffe6f9a647424beca"}
Sep 17 15:50:40.435622 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.435501    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"
Sep 17 15:50:40.493243 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.493207    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"]
Sep 17 15:50:40.499093 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.499063    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-f9f2bc-predictor-85cb9fd88f-z67ff"]
Sep 17 15:50:40.809063 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.809032    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"]
Sep 17 15:50:40.809405 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.809391    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1058b8a-af45-425b-80f7-a442c43d20e9" containerName="storage-initializer"
Sep 17 15:50:40.809495 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.809409    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1058b8a-af45-425b-80f7-a442c43d20e9" containerName="storage-initializer"
Sep 17 15:50:40.809495 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.809430    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1058b8a-af45-425b-80f7-a442c43d20e9" containerName="storage-initializer"
Sep 17 15:50:40.809495 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.809438    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1058b8a-af45-425b-80f7-a442c43d20e9" containerName="storage-initializer"
Sep 17 15:50:40.809597 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.809526    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1058b8a-af45-425b-80f7-a442c43d20e9" containerName="storage-initializer"
Sep 17 15:50:40.809652 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.809642    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1058b8a-af45-425b-80f7-a442c43d20e9" containerName="storage-initializer"
Sep 17 15:50:40.831572 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.831544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"]
Sep 17 15:50:40.831709 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.831686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:40.833413 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.833391    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 17 15:50:40.833703 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.833682    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 15:50:40.981729 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.981689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a422334-1cf1-44de-a541-7f99df441489-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:40.982177 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.981753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a422334-1cf1-44de-a541-7f99df441489-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:40.982177 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.981785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:40.982177 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.981802    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:40.982177 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:40.981914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5c7j\" (UniqueName: \"kubernetes.io/projected/8a422334-1cf1-44de-a541-7f99df441489-kube-api-access-t5c7j\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.083571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.083481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5c7j\" (UniqueName: \"kubernetes.io/projected/8a422334-1cf1-44de-a541-7f99df441489-kube-api-access-t5c7j\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.083571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.083559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a422334-1cf1-44de-a541-7f99df441489-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.083800 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.083609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a422334-1cf1-44de-a541-7f99df441489-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.083800 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.083648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.083800 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.083676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.084092 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.084068    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a422334-1cf1-44de-a541-7f99df441489-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.084367 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.084347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.084482 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.084406    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.086256 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.086235    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a422334-1cf1-44de-a541-7f99df441489-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.091682 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.091658    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5c7j\" (UniqueName: \"kubernetes.io/projected/8a422334-1cf1-44de-a541-7f99df441489-kube-api-access-t5c7j\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-sswfz\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.144532 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.144501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:50:41.558400 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:41.558363    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1058b8a-af45-425b-80f7-a442c43d20e9" path="/var/lib/kubelet/pods/d1058b8a-af45-425b-80f7-a442c43d20e9/volumes"
Sep 17 15:50:44.192032 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:44.191898    2581 scope.go:117] "RemoveContainer" containerID="f39959e75b56adaedb7f02736a92ae9f4cd36e41327aacaa9f467fffe850c6fc"
Sep 17 15:50:44.211565 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:44.211536    2581 scope.go:117] "RemoveContainer" containerID="5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81"
Sep 17 15:50:44.211949 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:50:44.211928    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81\": container with ID starting with 5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81 not found: ID does not exist" containerID="5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81"
Sep 17 15:50:44.212032 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:44.211968    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81"} err="failed to get container status \"5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81\": rpc error: code = NotFound desc = could not find container \"5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81\": container with ID starting with 5a30ac031fd07cde2274906ff265216b46fdb6adb33f7b134a9d361dff050a81 not found: ID does not exist"
Sep 17 15:50:44.458202 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:44.458129    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerStarted","Data":"d9608eec6dc466d5464a76c10f562842126064800ad4f1cf4b9f0cb82ea64055"}
Sep 17 15:50:44.466701 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:44.466678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"]
Sep 17 15:50:44.468936 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:50:44.468911    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a422334_1cf1_44de_a541_7f99df441489.slice/crio-a578a5e9f58f4068314308118480787173dd314fb116555d6660baeed4be74f9 WatchSource:0}: Error finding container a578a5e9f58f4068314308118480787173dd314fb116555d6660baeed4be74f9: Status 404 returned error can't find the container with id a578a5e9f58f4068314308118480787173dd314fb116555d6660baeed4be74f9
Sep 17 15:50:45.464869 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:45.464837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" event={"ID":"8a422334-1cf1-44de-a541-7f99df441489","Type":"ContainerStarted","Data":"50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d"}
Sep 17 15:50:45.465286 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:45.464898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" event={"ID":"8a422334-1cf1-44de-a541-7f99df441489","Type":"ContainerStarted","Data":"a578a5e9f58f4068314308118480787173dd314fb116555d6660baeed4be74f9"}
Sep 17 15:50:47.479224 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:47.479187    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerStarted","Data":"526a424c3cd33768de8338536fe018951a478978e2a1986f3e1a5ad6269ba596"}
Sep 17 15:50:49.490713 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:49.490673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerStarted","Data":"d9671a1505cbec531673d8a9e36d8f882f20e9c20208c94668312a1a921c704f"}
Sep 17 15:50:49.491154 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:49.490871    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:49.491154 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:49.491056    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:49.491154 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:49.491120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:49.492384 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:49.492363    2581 generic.go:356] "Generic (PLEG): container finished" podID="8a422334-1cf1-44de-a541-7f99df441489" containerID="50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d" exitCode=0
Sep 17 15:50:49.492530 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:49.492404    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" event={"ID":"8a422334-1cf1-44de-a541-7f99df441489","Type":"ContainerDied","Data":"50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d"}
Sep 17 15:50:49.492826 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:49.492782    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:50:49.493559 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:49.493502    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:50:49.517975 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:49.517926    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podStartSLOduration=3.253705152 podStartE2EDuration="24.517910682s" totalImagesPullingTime="17.244962117s" totalInitContainerRuntime="4.019243413s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:50:49.516877898 +0000 UTC m=+740.440452489" watchObservedRunningTime="2026-09-17 15:50:49.517910682 +0000 UTC m=+740.441485248"
Sep 17 15:50:50.502201 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:50.501766    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:50:50.503668 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:50.503554    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:50:55.509143 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:55.509108    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:50:55.510374 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:55.510214    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:50:55.510675 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:50:55.510640    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:05.510455 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:05.510383    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:51:05.510933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:05.510800    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:12.623369 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:12.623334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" event={"ID":"8a422334-1cf1-44de-a541-7f99df441489","Type":"ContainerStarted","Data":"96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67"}
Sep 17 15:51:12.623849 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:12.623398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" event={"ID":"8a422334-1cf1-44de-a541-7f99df441489","Type":"ContainerStarted","Data":"01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025"}
Sep 17 15:51:12.623849 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:12.623633    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:51:12.643252 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:12.643205    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podStartSLOduration=6.527830432 podStartE2EDuration="32.643187383s" totalImagesPullingTime="22.039298863s" totalInitContainerRuntime="4.076058088s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:51:12.641560504 +0000 UTC m=+763.565135076" watchObservedRunningTime="2026-09-17 15:51:12.643187383 +0000 UTC m=+763.566761951"
Sep 17 15:51:13.627647 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:13.627617    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:51:13.628851 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:13.628828    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:51:14.631400 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:14.631365    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:51:15.511095 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:15.511052    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:51:15.511621 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:15.511594    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:19.635578 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:19.635547    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:51:19.636084 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:19.636037    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:51:25.510300 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:25.510250    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:51:25.510754 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:25.510715    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:29.636067 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:29.636029    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:51:35.511041 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:35.510991    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:51:35.511447 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:35.511422    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:39.635978 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:39.635940    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:51:45.510651 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:45.510608    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:51:45.511121 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:45.511088    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:49.636220 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:49.636184    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:51:55.510406 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:55.510306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:51:55.510827 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:55.510800    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:51:59.636138 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:51:59.636103    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:52:02.553451 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:02.553406    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:52:02.553862 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:02.553729    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:09.636120 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:09.636081    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:52:12.554642 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:12.554609    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:52:12.555097 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:12.554926    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:52:19.636028 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:19.635985    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:52:20.816416 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:20.816369    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"]
Sep 17 15:52:20.816923 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:20.816685    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" containerID="cri-o://d9608eec6dc466d5464a76c10f562842126064800ad4f1cf4b9f0cb82ea64055" gracePeriod=30
Sep 17 15:52:20.816923 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:20.816790    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy" containerID="cri-o://526a424c3cd33768de8338536fe018951a478978e2a1986f3e1a5ad6269ba596" gracePeriod=30
Sep 17 15:52:20.816923 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:20.816808    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" containerID="cri-o://d9671a1505cbec531673d8a9e36d8f882f20e9c20208c94668312a1a921c704f" gracePeriod=30
Sep 17 15:52:20.937812 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:20.937786    2581 generic.go:356] "Generic (PLEG): container finished" podID="0d923b54-fbde-4925-80b3-587dd361db88" containerID="526a424c3cd33768de8338536fe018951a478978e2a1986f3e1a5ad6269ba596" exitCode=2
Sep 17 15:52:20.937941 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:20.937853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerDied","Data":"526a424c3cd33768de8338536fe018951a478978e2a1986f3e1a5ad6269ba596"}
Sep 17 15:52:21.037655 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.037625    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"]
Sep 17 15:52:21.041395 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.041379    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.044231 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.044212    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 17 15:52:21.044452 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.044436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 17 15:52:21.058515 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.058491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"]
Sep 17 15:52:21.085387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.085334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4907dc-28be-458c-b0c9-6930562a35c3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.085387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.085364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4907dc-28be-458c-b0c9-6930562a35c3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.085564 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.085407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.085564 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.085442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djpsj\" (UniqueName: \"kubernetes.io/projected/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-api-access-djpsj\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.085564 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.085487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.186119 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.186084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.186324 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.186128    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djpsj\" (UniqueName: \"kubernetes.io/projected/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-api-access-djpsj\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.186324 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.186177    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.186324 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.186237    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4907dc-28be-458c-b0c9-6930562a35c3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.186324 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.186269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4907dc-28be-458c-b0c9-6930562a35c3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.186809 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.186785    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4907dc-28be-458c-b0c9-6930562a35c3-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.187029 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.187006    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.187075 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.187056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.188724 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.188703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4907dc-28be-458c-b0c9-6930562a35c3-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.195107 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.195090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djpsj\" (UniqueName: \"kubernetes.io/projected/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-api-access-djpsj\") pod \"isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.351674 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.351598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:21.643246 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.643164    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"]
Sep 17 15:52:21.646055 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:52:21.646023    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4907dc_28be_458c_b0c9_6930562a35c3.slice/crio-062135de5e6cd393abe6c3a717e011881a7ace05c7d839f8a595110651337c98 WatchSource:0}: Error finding container 062135de5e6cd393abe6c3a717e011881a7ace05c7d839f8a595110651337c98: Status 404 returned error can't find the container with id 062135de5e6cd393abe6c3a717e011881a7ace05c7d839f8a595110651337c98
Sep 17 15:52:21.946824 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.946748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerStarted","Data":"22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e"}
Sep 17 15:52:21.947178 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:21.946830    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerStarted","Data":"062135de5e6cd393abe6c3a717e011881a7ace05c7d839f8a595110651337c98"}
Sep 17 15:52:22.553430 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:22.553390    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:52:22.553711 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:22.553684    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:25.502187 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:25.502145    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 15:52:25.969664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:25.969572    2581 generic.go:356] "Generic (PLEG): container finished" podID="0d923b54-fbde-4925-80b3-587dd361db88" containerID="d9608eec6dc466d5464a76c10f562842126064800ad4f1cf4b9f0cb82ea64055" exitCode=0
Sep 17 15:52:25.969664 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:25.969647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerDied","Data":"d9608eec6dc466d5464a76c10f562842126064800ad4f1cf4b9f0cb82ea64055"}
Sep 17 15:52:25.971193 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:25.971168    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerID="22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e" exitCode=0
Sep 17 15:52:25.971301 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:25.971243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerDied","Data":"22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e"}
Sep 17 15:52:26.977952 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:26.977915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerStarted","Data":"a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7"}
Sep 17 15:52:26.978352 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:26.977987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerStarted","Data":"d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e"}
Sep 17 15:52:26.978352 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:26.978018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerStarted","Data":"8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac"}
Sep 17 15:52:26.978352 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:26.978267    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:26.978352 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:26.978340    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:26.978560 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:26.978379    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:26.979853 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:26.979827    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:52:26.980532 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:26.980508    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:27.004572 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:27.004515    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podStartSLOduration=2.136529016 podStartE2EDuration="6.00449855s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.867969534s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:52:27.002177262 +0000 UTC m=+837.925751842" watchObservedRunningTime="2026-09-17 15:52:27.00449855 +0000 UTC m=+837.928073118"
Sep 17 15:52:27.982387 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:27.982346    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:52:27.982855 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:27.982831    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:29.636153 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:29.636119    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Sep 17 15:52:30.501737 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:30.501696    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 15:52:32.553540 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:32.553491    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:52:32.553990 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:32.553701    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:32.986518 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:32.986421    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:52:32.987008 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:32.986983    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:52:32.987291 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:32.987264    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:35.501883 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:35.501839    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 15:52:35.502259 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:35.501972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:52:39.636571 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:39.636539    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:52:40.502082 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:40.502037    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 15:52:41.427750 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:41.427719    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"]
Sep 17 15:52:41.428154 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:41.427960    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container" containerID="cri-o://01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025" gracePeriod=30
Sep 17 15:52:41.428154 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:41.428010    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kube-rbac-proxy" containerID="cri-o://96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67" gracePeriod=30
Sep 17 15:52:42.051066 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:42.051034    2581 generic.go:356] "Generic (PLEG): container finished" podID="8a422334-1cf1-44de-a541-7f99df441489" containerID="96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67" exitCode=2
Sep 17 15:52:42.051255 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:42.051110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" event={"ID":"8a422334-1cf1-44de-a541-7f99df441489","Type":"ContainerDied","Data":"96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67"}
Sep 17 15:52:42.554055 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:42.554015    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Sep 17 15:52:42.554520 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:42.554143    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:52:42.554520 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:42.554314    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:42.554520 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:42.554404    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:52:42.986998 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:42.986904    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:52:42.987314 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:42.987285    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:44.632320 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:44.632276    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Sep 17 15:52:45.502036 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:45.501979    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 15:52:46.774051 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.774028    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:52:46.915758 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.915658    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a422334-1cf1-44de-a541-7f99df441489-proxy-tls\") pod \"8a422334-1cf1-44de-a541-7f99df441489\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") "
Sep 17 15:52:46.915758 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.915750    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-kube-rbac-proxy-sar-config\") pod \"8a422334-1cf1-44de-a541-7f99df441489\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") "
Sep 17 15:52:46.916005 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.915791    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a422334-1cf1-44de-a541-7f99df441489-kserve-provision-location\") pod \"8a422334-1cf1-44de-a541-7f99df441489\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") "
Sep 17 15:52:46.916005 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.915847    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5c7j\" (UniqueName: \"kubernetes.io/projected/8a422334-1cf1-44de-a541-7f99df441489-kube-api-access-t5c7j\") pod \"8a422334-1cf1-44de-a541-7f99df441489\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") "
Sep 17 15:52:46.916005 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.915873    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-cabundle-cert\") pod \"8a422334-1cf1-44de-a541-7f99df441489\" (UID: \"8a422334-1cf1-44de-a541-7f99df441489\") "
Sep 17 15:52:46.916192 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.916169    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a422334-1cf1-44de-a541-7f99df441489-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8a422334-1cf1-44de-a541-7f99df441489" (UID: "8a422334-1cf1-44de-a541-7f99df441489"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:52:46.916296 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.916217    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8a422334-1cf1-44de-a541-7f99df441489" (UID: "8a422334-1cf1-44de-a541-7f99df441489"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:52:46.916350 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.916291    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8a422334-1cf1-44de-a541-7f99df441489" (UID: "8a422334-1cf1-44de-a541-7f99df441489"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:52:46.917907 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.917886    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a422334-1cf1-44de-a541-7f99df441489-kube-api-access-t5c7j" (OuterVolumeSpecName: "kube-api-access-t5c7j") pod "8a422334-1cf1-44de-a541-7f99df441489" (UID: "8a422334-1cf1-44de-a541-7f99df441489"). InnerVolumeSpecName "kube-api-access-t5c7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:52:46.918197 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:46.918175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a422334-1cf1-44de-a541-7f99df441489-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8a422334-1cf1-44de-a541-7f99df441489" (UID: "8a422334-1cf1-44de-a541-7f99df441489"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:52:47.017216 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.017186    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:47.017216 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.017211    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8a422334-1cf1-44de-a541-7f99df441489-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:47.017216 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.017224    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5c7j\" (UniqueName: \"kubernetes.io/projected/8a422334-1cf1-44de-a541-7f99df441489-kube-api-access-t5c7j\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:47.017459 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.017237    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8a422334-1cf1-44de-a541-7f99df441489-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:47.017459 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.017251    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8a422334-1cf1-44de-a541-7f99df441489-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:47.076340 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.076304    2581 generic.go:356] "Generic (PLEG): container finished" podID="8a422334-1cf1-44de-a541-7f99df441489" containerID="01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025" exitCode=0
Sep 17 15:52:47.076498 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.076364    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"
Sep 17 15:52:47.076498 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.076389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" event={"ID":"8a422334-1cf1-44de-a541-7f99df441489","Type":"ContainerDied","Data":"01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025"}
Sep 17 15:52:47.076498 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.076448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz" event={"ID":"8a422334-1cf1-44de-a541-7f99df441489","Type":"ContainerDied","Data":"a578a5e9f58f4068314308118480787173dd314fb116555d6660baeed4be74f9"}
Sep 17 15:52:47.076498 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.076493    2581 scope.go:117] "RemoveContainer" containerID="96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67"
Sep 17 15:52:47.088096 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.088074    2581 scope.go:117] "RemoveContainer" containerID="01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025"
Sep 17 15:52:47.095720 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.095700    2581 scope.go:117] "RemoveContainer" containerID="50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d"
Sep 17 15:52:47.103078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.102963    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"]
Sep 17 15:52:47.103078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.103067    2581 scope.go:117] "RemoveContainer" containerID="96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67"
Sep 17 15:52:47.103350 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:52:47.103330    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67\": container with ID starting with 96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67 not found: ID does not exist" containerID="96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67"
Sep 17 15:52:47.103394 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.103358    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67"} err="failed to get container status \"96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67\": rpc error: code = NotFound desc = could not find container \"96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67\": container with ID starting with 96ab9d6774e4382e2bf89661cb509271a1bf58d5affa042cd7ee9290122e0c67 not found: ID does not exist"
Sep 17 15:52:47.103394 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.103376    2581 scope.go:117] "RemoveContainer" containerID="01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025"
Sep 17 15:52:47.103647 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:52:47.103628    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025\": container with ID starting with 01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025 not found: ID does not exist" containerID="01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025"
Sep 17 15:52:47.103724 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.103655    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025"} err="failed to get container status \"01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025\": rpc error: code = NotFound desc = could not find container \"01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025\": container with ID starting with 01816967a7203608cf7ff1b9cb73404a49f1ba1da4664c0f82ad425f15fad025 not found: ID does not exist"
Sep 17 15:52:47.103724 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.103677    2581 scope.go:117] "RemoveContainer" containerID="50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d"
Sep 17 15:52:47.104037 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:52:47.104017    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d\": container with ID starting with 50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d not found: ID does not exist" containerID="50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d"
Sep 17 15:52:47.104158 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.104044    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d"} err="failed to get container status \"50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d\": rpc error: code = NotFound desc = could not find container \"50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d\": container with ID starting with 50afe15776295e3120dde2f993a4c564b4f78d6c3342e59e4c95dbe2b94ede9d not found: ID does not exist"
Sep 17 15:52:47.105668 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.105650    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-sswfz"]
Sep 17 15:52:47.557306 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:47.557265    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a422334-1cf1-44de-a541-7f99df441489" path="/var/lib/kubelet/pods/8a422334-1cf1-44de-a541-7f99df441489/volumes"
Sep 17 15:52:50.502367 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:50.502322    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Sep 17 15:52:51.097574 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.097495    2581 generic.go:356] "Generic (PLEG): container finished" podID="0d923b54-fbde-4925-80b3-587dd361db88" containerID="d9671a1505cbec531673d8a9e36d8f882f20e9c20208c94668312a1a921c704f" exitCode=0
Sep 17 15:52:51.097574 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.097543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerDied","Data":"d9671a1505cbec531673d8a9e36d8f882f20e9c20208c94668312a1a921c704f"}
Sep 17 15:52:51.461285 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.461254    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:52:51.554007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.553975    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d923b54-fbde-4925-80b3-587dd361db88-proxy-tls\") pod \"0d923b54-fbde-4925-80b3-587dd361db88\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") "
Sep 17 15:52:51.554492 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.554028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d923b54-fbde-4925-80b3-587dd361db88-kserve-provision-location\") pod \"0d923b54-fbde-4925-80b3-587dd361db88\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") "
Sep 17 15:52:51.554492 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.554070    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-kube-rbac-proxy-sar-config\") pod \"0d923b54-fbde-4925-80b3-587dd361db88\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") "
Sep 17 15:52:51.554492 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.554093    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25d28\" (UniqueName: \"kubernetes.io/projected/0d923b54-fbde-4925-80b3-587dd361db88-kube-api-access-25d28\") pod \"0d923b54-fbde-4925-80b3-587dd361db88\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") "
Sep 17 15:52:51.554492 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.554119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-cabundle-cert\") pod \"0d923b54-fbde-4925-80b3-587dd361db88\" (UID: \"0d923b54-fbde-4925-80b3-587dd361db88\") "
Sep 17 15:52:51.554492 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.554404    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d923b54-fbde-4925-80b3-587dd361db88-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0d923b54-fbde-4925-80b3-587dd361db88" (UID: "0d923b54-fbde-4925-80b3-587dd361db88"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:52:51.554714 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.554563    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d923b54-fbde-4925-80b3-587dd361db88" (UID: "0d923b54-fbde-4925-80b3-587dd361db88"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:52:51.554714 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.554572    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0d923b54-fbde-4925-80b3-587dd361db88" (UID: "0d923b54-fbde-4925-80b3-587dd361db88"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:52:51.556164 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.556141    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d923b54-fbde-4925-80b3-587dd361db88-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d923b54-fbde-4925-80b3-587dd361db88" (UID: "0d923b54-fbde-4925-80b3-587dd361db88"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:52:51.556459 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.556439    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d923b54-fbde-4925-80b3-587dd361db88-kube-api-access-25d28" (OuterVolumeSpecName: "kube-api-access-25d28") pod "0d923b54-fbde-4925-80b3-587dd361db88" (UID: "0d923b54-fbde-4925-80b3-587dd361db88"). InnerVolumeSpecName "kube-api-access-25d28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:52:51.656135 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.656028    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d923b54-fbde-4925-80b3-587dd361db88-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:51.656135 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.656066    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0d923b54-fbde-4925-80b3-587dd361db88-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:51.656135 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.656083    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:51.656135 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.656095    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25d28\" (UniqueName: \"kubernetes.io/projected/0d923b54-fbde-4925-80b3-587dd361db88-kube-api-access-25d28\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:51.656135 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:51.656109    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0d923b54-fbde-4925-80b3-587dd361db88-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:52:52.104500 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.104445    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c" event={"ID":"0d923b54-fbde-4925-80b3-587dd361db88","Type":"ContainerDied","Data":"46f6736ee72af2cb07e6777f58950e9fc690e2db96439df59d22e0723120d567"}
Sep 17 15:52:52.104500 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.104491    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"
Sep 17 15:52:52.104708 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.104537    2581 scope.go:117] "RemoveContainer" containerID="d9671a1505cbec531673d8a9e36d8f882f20e9c20208c94668312a1a921c704f"
Sep 17 15:52:52.112661 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.112647    2581 scope.go:117] "RemoveContainer" containerID="526a424c3cd33768de8338536fe018951a478978e2a1986f3e1a5ad6269ba596"
Sep 17 15:52:52.120867 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.120844    2581 scope.go:117] "RemoveContainer" containerID="d9608eec6dc466d5464a76c10f562842126064800ad4f1cf4b9f0cb82ea64055"
Sep 17 15:52:52.126992 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.126968    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"]
Sep 17 15:52:52.128535 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.128516    2581 scope.go:117] "RemoveContainer" containerID="25ceb744b320043d47fc5f5ec13facc071686496373662e4b075ef5c7698b852"
Sep 17 15:52:52.130113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.130095    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-6c74cc89d5-km88c"]
Sep 17 15:52:52.987772 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.987733    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:52:52.988246 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:52.988222    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:52:53.557365 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:52:53.557329    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d923b54-fbde-4925-80b3-587dd361db88" path="/var/lib/kubelet/pods/0d923b54-fbde-4925-80b3-587dd361db88/volumes"
Sep 17 15:53:02.987229 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:02.987183    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:53:02.987702 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:02.987678    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:53:12.987268 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:12.987219    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:53:12.987716 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:12.987689    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:53:22.987517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:22.987398    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:53:22.987919 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:22.987840    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:53:32.987379 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:32.987336    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:53:32.987858 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:32.987783    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:53:40.553764 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:40.553728    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:53:40.554137 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:40.553971    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:53:46.113103 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.113070    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"]
Sep 17 15:53:46.113515 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.113355    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" containerID="cri-o://8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac" gracePeriod=30
Sep 17 15:53:46.113515 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.113382    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy" containerID="cri-o://d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e" gracePeriod=30
Sep 17 15:53:46.113515 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.113399    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" containerID="cri-o://a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7" gracePeriod=30
Sep 17 15:53:46.240572 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240540    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"]
Sep 17 15:53:46.240913 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240900    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="storage-initializer"
Sep 17 15:53:46.240965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240914    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="storage-initializer"
Sep 17 15:53:46.240965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240929    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent"
Sep 17 15:53:46.240965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240935    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent"
Sep 17 15:53:46.240965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240942    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy"
Sep 17 15:53:46.240965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240948    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy"
Sep 17 15:53:46.240965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240957    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kube-rbac-proxy"
Sep 17 15:53:46.240965 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240962    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kube-rbac-proxy"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240969    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="storage-initializer"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240975    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="storage-initializer"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240984    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.240989    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.241000    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.241005    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.241054    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="agent"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.241060    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kube-rbac-proxy"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.241069    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a422334-1cf1-44de-a541-7f99df441489" containerName="kserve-container"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.241076    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kserve-container"
Sep 17 15:53:46.241191 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.241080    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d923b54-fbde-4925-80b3-587dd361db88" containerName="kube-rbac-proxy"
Sep 17 15:53:46.244103 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.244082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.246146 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.246125    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Sep 17 15:53:46.246235 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.246125    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Sep 17 15:53:46.262573 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.262548    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"]
Sep 17 15:53:46.328324 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.328287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.328511 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.328344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcrtg\" (UniqueName: \"kubernetes.io/projected/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-api-access-vcrtg\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.328511 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.328386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.356531 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.356497    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerID="d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e" exitCode=2
Sep 17 15:53:46.356659 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.356535    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerDied","Data":"d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e"}
Sep 17 15:53:46.429599 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.429511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcrtg\" (UniqueName: \"kubernetes.io/projected/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-api-access-vcrtg\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.429599 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.429553    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.429599 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.429598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.429868 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:53:46.429745    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Sep 17 15:53:46.429868 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:53:46.429816    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-proxy-tls podName:cb3faea3-47f6-4be2-9cc7-607c84e60e3d nodeName:}" failed. No retries permitted until 2026-09-17 15:53:46.929795434 +0000 UTC m=+917.853369993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-5kzf7" (UID: "cb3faea3-47f6-4be2-9cc7-607c84e60e3d") : secret "message-dumper-predictor-serving-cert" not found
Sep 17 15:53:46.430245 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.430227    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.442511 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.442488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcrtg\" (UniqueName: \"kubernetes.io/projected/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-api-access-vcrtg\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.934585 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.934546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:46.936928 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:46.936908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-5kzf7\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:47.153139 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:47.153100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:47.442011 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:47.441980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"]
Sep 17 15:53:47.445594 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:53:47.445569    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3faea3_47f6_4be2_9cc7_607c84e60e3d.slice/crio-7be48b222cfed08d54d4b5b3dc5b25dacb0920b2f9c82fa26c73fba7c72402e9 WatchSource:0}: Error finding container 7be48b222cfed08d54d4b5b3dc5b25dacb0920b2f9c82fa26c73fba7c72402e9: Status 404 returned error can't find the container with id 7be48b222cfed08d54d4b5b3dc5b25dacb0920b2f9c82fa26c73fba7c72402e9
Sep 17 15:53:47.447694 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:47.447677    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 15:53:47.983424 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:47.983382    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 15:53:48.368342 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:48.368296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7" event={"ID":"cb3faea3-47f6-4be2-9cc7-607c84e60e3d","Type":"ContainerStarted","Data":"7be48b222cfed08d54d4b5b3dc5b25dacb0920b2f9c82fa26c73fba7c72402e9"}
Sep 17 15:53:49.375386 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:49.375351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7" event={"ID":"cb3faea3-47f6-4be2-9cc7-607c84e60e3d","Type":"ContainerStarted","Data":"793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8"}
Sep 17 15:53:49.375810 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:49.375414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:49.375810 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:49.375443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7" event={"ID":"cb3faea3-47f6-4be2-9cc7-607c84e60e3d","Type":"ContainerStarted","Data":"42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7"}
Sep 17 15:53:49.375810 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:49.375483    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:49.377256 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:49.377236    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:49.416023 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:49.415967    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7" podStartSLOduration=2.0651865 podStartE2EDuration="3.415947025s" totalImagesPullingTime="1.350760525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:53:49.414975849 +0000 UTC m=+920.338550546" watchObservedRunningTime="2026-09-17 15:53:49.415947025 +0000 UTC m=+920.339521600"
Sep 17 15:53:50.553665 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:50.553614    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:53:50.554052 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:50.553933    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:53:51.386984 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:51.386952    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerID="8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac" exitCode=0
Sep 17 15:53:51.387179 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:51.387028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerDied","Data":"8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac"}
Sep 17 15:53:52.982948 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:52.982904    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 15:53:56.391433 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:56.391405    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:53:57.983624 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:57.983578    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 15:53:57.984041 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:53:57.983721    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:54:00.553129 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:00.553085    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:54:00.553588 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:00.553432    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:54:02.982792 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:02.982748    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 15:54:06.291812 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.291778    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"]
Sep 17 15:54:06.298086 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.298069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.300033 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.300013    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 17 15:54:06.300128 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.300051    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 17 15:54:06.312445 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.312420    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"]
Sep 17 15:54:06.404734 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.404697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.404918 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.404758    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-cabundle-cert\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.404918 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.404784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxk4\" (UniqueName: \"kubernetes.io/projected/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-api-access-lnxk4\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.404918 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.404803    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad73a291-22e3-4802-91a4-a2bc9c3c111e-proxy-tls\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.405032 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.404930    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kserve-provision-location\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.506022 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.505983    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-cabundle-cert\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.506022 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.506031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxk4\" (UniqueName: \"kubernetes.io/projected/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-api-access-lnxk4\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.506292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.506051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad73a291-22e3-4802-91a4-a2bc9c3c111e-proxy-tls\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.506292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.506088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kserve-provision-location\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.506292 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.506115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.506567 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.506538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kserve-provision-location\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.506811 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.506793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-cabundle-cert\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.506875 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.506838    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.508415 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.508398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad73a291-22e3-4802-91a4-a2bc9c3c111e-proxy-tls\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.519079 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.519056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxk4\" (UniqueName: \"kubernetes.io/projected/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-api-access-lnxk4\") pod \"isvc-logger-predictor-76485c66c8-92wfn\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.607196 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.607039    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:06.891125 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:06.891102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"]
Sep 17 15:54:06.893162 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:54:06.893129    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad73a291_22e3_4802_91a4_a2bc9c3c111e.slice/crio-72130ea46d42ddc4e88090c3e33c8a47c9e2cb930bac36b561e7cc57269456d8 WatchSource:0}: Error finding container 72130ea46d42ddc4e88090c3e33c8a47c9e2cb930bac36b561e7cc57269456d8: Status 404 returned error can't find the container with id 72130ea46d42ddc4e88090c3e33c8a47c9e2cb930bac36b561e7cc57269456d8
Sep 17 15:54:07.463287 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:07.463251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerStarted","Data":"138d0974b865608a1702c3ed28e50043a788b4090687e021e15de828fc42da94"}
Sep 17 15:54:07.463715 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:07.463347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerStarted","Data":"72130ea46d42ddc4e88090c3e33c8a47c9e2cb930bac36b561e7cc57269456d8"}
Sep 17 15:54:07.983069 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:07.983021    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 15:54:10.553189 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:10.553148    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Sep 17 15:54:10.553660 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:10.553288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:54:10.553660 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:10.553455    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:54:10.553660 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:10.553606    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:54:11.480883 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:11.480848    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerID="138d0974b865608a1702c3ed28e50043a788b4090687e021e15de828fc42da94" exitCode=0
Sep 17 15:54:11.481056 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:11.480918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerDied","Data":"138d0974b865608a1702c3ed28e50043a788b4090687e021e15de828fc42da94"}
Sep 17 15:54:12.488011 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.487975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerStarted","Data":"e3eaa8fac7e7304730a0c23ed06dc89d8b5838f6028f798c8563180163123d7d"}
Sep 17 15:54:12.488439 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.488074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerStarted","Data":"3a97bee7d119d0f4fefeb52c52957c2d8fdff2b14f47b235d4e6ff0e86c7d6c0"}
Sep 17 15:54:12.488439 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.488099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerStarted","Data":"d0b9da547738ad1e397ae75eb73d93def3c94f526d555cd0728d744477b12233"}
Sep 17 15:54:12.488439 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.488397    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:12.488609 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.488480    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:12.488609 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.488520    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:12.489908 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.489876    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:54:12.490603 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.490578    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:54:12.535750 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.535699    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podStartSLOduration=2.48380464 podStartE2EDuration="6.535685009s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.051880369s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:54:12.534627937 +0000 UTC m=+943.458202515" watchObservedRunningTime="2026-09-17 15:54:12.535685009 +0000 UTC m=+943.459259612"
Sep 17 15:54:12.983247 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:12.983203    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Sep 17 15:54:13.493316 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:13.493277    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:54:13.493855 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:13.493719    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:54:16.252668 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.252641    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:54:16.393689 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.393611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-cabundle-cert\") pod \"4b4907dc-28be-458c-b0c9-6930562a35c3\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") "
Sep 17 15:54:16.393689 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.393648    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djpsj\" (UniqueName: \"kubernetes.io/projected/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-api-access-djpsj\") pod \"4b4907dc-28be-458c-b0c9-6930562a35c3\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") "
Sep 17 15:54:16.393912 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.393717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4907dc-28be-458c-b0c9-6930562a35c3-kserve-provision-location\") pod \"4b4907dc-28be-458c-b0c9-6930562a35c3\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") "
Sep 17 15:54:16.393912 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.393740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4907dc-28be-458c-b0c9-6930562a35c3-proxy-tls\") pod \"4b4907dc-28be-458c-b0c9-6930562a35c3\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") "
Sep 17 15:54:16.393912 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.393756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-rbac-proxy-sar-config\") pod \"4b4907dc-28be-458c-b0c9-6930562a35c3\" (UID: \"4b4907dc-28be-458c-b0c9-6930562a35c3\") "
Sep 17 15:54:16.394069 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.394000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b4907dc-28be-458c-b0c9-6930562a35c3" (UID: "4b4907dc-28be-458c-b0c9-6930562a35c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:16.394069 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.394038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4907dc-28be-458c-b0c9-6930562a35c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b4907dc-28be-458c-b0c9-6930562a35c3" (UID: "4b4907dc-28be-458c-b0c9-6930562a35c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:54:16.394143 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.394101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b4907dc-28be-458c-b0c9-6930562a35c3" (UID: "4b4907dc-28be-458c-b0c9-6930562a35c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:54:16.395813 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.395794    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b4907dc-28be-458c-b0c9-6930562a35c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b4907dc-28be-458c-b0c9-6930562a35c3" (UID: "4b4907dc-28be-458c-b0c9-6930562a35c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:54:16.395908 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.395834    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-api-access-djpsj" (OuterVolumeSpecName: "kube-api-access-djpsj") pod "4b4907dc-28be-458c-b0c9-6930562a35c3" (UID: "4b4907dc-28be-458c-b0c9-6930562a35c3"). InnerVolumeSpecName "kube-api-access-djpsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:54:16.495289 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.495235    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b4907dc-28be-458c-b0c9-6930562a35c3-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:54:16.495289 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.495278    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b4907dc-28be-458c-b0c9-6930562a35c3-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:54:16.495289 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.495291    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:54:16.495289 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.495301    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b4907dc-28be-458c-b0c9-6930562a35c3-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:54:16.495289 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.495310    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djpsj\" (UniqueName: \"kubernetes.io/projected/4b4907dc-28be-458c-b0c9-6930562a35c3-kube-api-access-djpsj\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:54:16.507962 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.507934    2581 generic.go:356] "Generic (PLEG): container finished" podID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerID="a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7" exitCode=0
Sep 17 15:54:16.508085 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.507993    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"
Sep 17 15:54:16.508085 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.508021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerDied","Data":"a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7"}
Sep 17 15:54:16.508211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.508102    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v" event={"ID":"4b4907dc-28be-458c-b0c9-6930562a35c3","Type":"ContainerDied","Data":"062135de5e6cd393abe6c3a717e011881a7ace05c7d839f8a595110651337c98"}
Sep 17 15:54:16.508211 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.508149    2581 scope.go:117] "RemoveContainer" containerID="a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7"
Sep 17 15:54:16.516796 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.516776    2581 scope.go:117] "RemoveContainer" containerID="d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e"
Sep 17 15:54:16.524721 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.524704    2581 scope.go:117] "RemoveContainer" containerID="8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac"
Sep 17 15:54:16.531680 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.531638    2581 scope.go:117] "RemoveContainer" containerID="22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e"
Sep 17 15:54:16.533979 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.533959    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"]
Sep 17 15:54:16.539030 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.539014    2581 scope.go:117] "RemoveContainer" containerID="a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7"
Sep 17 15:54:16.539358 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:54:16.539331    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7\": container with ID starting with a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7 not found: ID does not exist" containerID="a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7"
Sep 17 15:54:16.539593 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.539369    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7"} err="failed to get container status \"a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7\": rpc error: code = NotFound desc = could not find container \"a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7\": container with ID starting with a12c4e63699a745775d040fa7ff20f863a2058bdf500855d314689cf22469bc7 not found: ID does not exist"
Sep 17 15:54:16.539593 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.539393    2581 scope.go:117] "RemoveContainer" containerID="d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e"
Sep 17 15:54:16.539809 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:54:16.539786    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e\": container with ID starting with d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e not found: ID does not exist" containerID="d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e"
Sep 17 15:54:16.539872 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.539827    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e"} err="failed to get container status \"d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e\": rpc error: code = NotFound desc = could not find container \"d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e\": container with ID starting with d46038b7991b0cf320b776cce3faa770ef837f6aced053d7cd727bcc118c6d3e not found: ID does not exist"
Sep 17 15:54:16.539872 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.539850    2581 scope.go:117] "RemoveContainer" containerID="8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac"
Sep 17 15:54:16.540218 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:54:16.540197    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac\": container with ID starting with 8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac not found: ID does not exist" containerID="8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac"
Sep 17 15:54:16.540306 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.540221    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac"} err="failed to get container status \"8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac\": rpc error: code = NotFound desc = could not find container \"8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac\": container with ID starting with 8991dcc9314a046097bc4efcc1cf8eb4416e5a2d7e69c9e04bf56f4f46a51eac not found: ID does not exist"
Sep 17 15:54:16.540306 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.540237    2581 scope.go:117] "RemoveContainer" containerID="22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e"
Sep 17 15:54:16.540764 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:54:16.540739    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e\": container with ID starting with 22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e not found: ID does not exist" containerID="22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e"
Sep 17 15:54:16.540835 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.540771    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e"} err="failed to get container status \"22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e\": rpc error: code = NotFound desc = could not find container \"22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e\": container with ID starting with 22e34a4eb3aee1ab8e631b1de114dbd965dca81245e247164c9b74b402f1012e not found: ID does not exist"
Sep 17 15:54:16.541514 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:16.541496    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-76bd9f8cf9-7b49v"]
Sep 17 15:54:17.558046 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:17.558013    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" path="/var/lib/kubelet/pods/4b4907dc-28be-458c-b0c9-6930562a35c3/volumes"
Sep 17 15:54:18.497530 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:18.497500    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:54:18.498041 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:18.498005    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:54:18.498339 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:18.498317    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:54:28.498044 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:28.497987    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:54:28.498621 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:28.498443    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:54:38.498756 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:38.498710    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:54:38.499184 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:38.499155    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:54:48.498710 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:48.498657    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:54:48.499103 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:48.499053    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:54:58.498690 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:58.498604    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:54:58.499257 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:54:58.499234    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:55:08.497967 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:08.497915    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:55:08.498442 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:08.498350    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:55:18.498439 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:18.498396    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:55:18.498895 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:18.498872    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:55:28.498707 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:28.498672    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:55:28.499213 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:28.499195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:55:41.200542 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.200509    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-5kzf7_cb3faea3-47f6-4be2-9cc7-607c84e60e3d/kserve-container/0.log"
Sep 17 15:55:41.378732 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.378701    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"]
Sep 17 15:55:41.379018 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.378987    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" containerID="cri-o://d0b9da547738ad1e397ae75eb73d93def3c94f526d555cd0728d744477b12233" gracePeriod=30
Sep 17 15:55:41.379258 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.378989    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" containerID="cri-o://e3eaa8fac7e7304730a0c23ed06dc89d8b5838f6028f798c8563180163123d7d" gracePeriod=30
Sep 17 15:55:41.379369 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.379019    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy" containerID="cri-o://3a97bee7d119d0f4fefeb52c52957c2d8fdff2b14f47b235d4e6ff0e86c7d6c0" gracePeriod=30
Sep 17 15:55:41.408702 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.408671    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"]
Sep 17 15:55:41.408925 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.408898    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7" podUID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerName="kserve-container" containerID="cri-o://42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7" gracePeriod=30
Sep 17 15:55:41.409034 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.409003    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7" podUID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerName="kube-rbac-proxy" containerID="cri-o://793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8" gracePeriod=30
Sep 17 15:55:41.651772 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.651747    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:55:41.701244 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.701211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcrtg\" (UniqueName: \"kubernetes.io/projected/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-api-access-vcrtg\") pod \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") "
Sep 17 15:55:41.701398 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.701283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-proxy-tls\") pod \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") "
Sep 17 15:55:41.701398 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.701314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-rbac-proxy-sar-config\") pod \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\" (UID: \"cb3faea3-47f6-4be2-9cc7-607c84e60e3d\") "
Sep 17 15:55:41.701706 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.701678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cb3faea3-47f6-4be2-9cc7-607c84e60e3d" (UID: "cb3faea3-47f6-4be2-9cc7-607c84e60e3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:55:41.703419 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.703397    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-api-access-vcrtg" (OuterVolumeSpecName: "kube-api-access-vcrtg") pod "cb3faea3-47f6-4be2-9cc7-607c84e60e3d" (UID: "cb3faea3-47f6-4be2-9cc7-607c84e60e3d"). InnerVolumeSpecName "kube-api-access-vcrtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:55:41.703507 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.703423    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cb3faea3-47f6-4be2-9cc7-607c84e60e3d" (UID: "cb3faea3-47f6-4be2-9cc7-607c84e60e3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:55:41.802454 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.802423    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcrtg\" (UniqueName: \"kubernetes.io/projected/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-api-access-vcrtg\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:55:41.802454 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.802448    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:55:41.802454 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.802458    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cb3faea3-47f6-4be2-9cc7-607c84e60e3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:55:41.869448 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.869416    2581 generic.go:356] "Generic (PLEG): container finished" podID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerID="793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8" exitCode=2
Sep 17 15:55:41.869448 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.869441    2581 generic.go:356] "Generic (PLEG): container finished" podID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerID="42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7" exitCode=2
Sep 17 15:55:41.869686 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.869498    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"
Sep 17 15:55:41.869686 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.869498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7" event={"ID":"cb3faea3-47f6-4be2-9cc7-607c84e60e3d","Type":"ContainerDied","Data":"793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8"}
Sep 17 15:55:41.869686 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.869644    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7" event={"ID":"cb3faea3-47f6-4be2-9cc7-607c84e60e3d","Type":"ContainerDied","Data":"42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7"}
Sep 17 15:55:41.869686 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.869667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7" event={"ID":"cb3faea3-47f6-4be2-9cc7-607c84e60e3d","Type":"ContainerDied","Data":"7be48b222cfed08d54d4b5b3dc5b25dacb0920b2f9c82fa26c73fba7c72402e9"}
Sep 17 15:55:41.869839 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.869691    2581 scope.go:117] "RemoveContainer" containerID="793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8"
Sep 17 15:55:41.872385 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.872366    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerID="3a97bee7d119d0f4fefeb52c52957c2d8fdff2b14f47b235d4e6ff0e86c7d6c0" exitCode=2
Sep 17 15:55:41.872559 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.872432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerDied","Data":"3a97bee7d119d0f4fefeb52c52957c2d8fdff2b14f47b235d4e6ff0e86c7d6c0"}
Sep 17 15:55:41.878273 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.878255    2581 scope.go:117] "RemoveContainer" containerID="42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7"
Sep 17 15:55:41.885821 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.885806    2581 scope.go:117] "RemoveContainer" containerID="793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8"
Sep 17 15:55:41.886087 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:55:41.886065    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8\": container with ID starting with 793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8 not found: ID does not exist" containerID="793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8"
Sep 17 15:55:41.886152 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.886092    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8"} err="failed to get container status \"793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8\": rpc error: code = NotFound desc = could not find container \"793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8\": container with ID starting with 793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8 not found: ID does not exist"
Sep 17 15:55:41.886152 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.886106    2581 scope.go:117] "RemoveContainer" containerID="42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7"
Sep 17 15:55:41.886314 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:55:41.886295    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7\": container with ID starting with 42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7 not found: ID does not exist" containerID="42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7"
Sep 17 15:55:41.886361 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.886320    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7"} err="failed to get container status \"42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7\": rpc error: code = NotFound desc = could not find container \"42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7\": container with ID starting with 42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7 not found: ID does not exist"
Sep 17 15:55:41.886361 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.886332    2581 scope.go:117] "RemoveContainer" containerID="793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8"
Sep 17 15:55:41.886586 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.886567    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8"} err="failed to get container status \"793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8\": rpc error: code = NotFound desc = could not find container \"793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8\": container with ID starting with 793683ca0ecf84037dfdd1bc156acc1271c838f8ca640e3208a5b36e7e6b27e8 not found: ID does not exist"
Sep 17 15:55:41.886586 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.886585    2581 scope.go:117] "RemoveContainer" containerID="42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7"
Sep 17 15:55:41.886764 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.886749    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7"} err="failed to get container status \"42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7\": rpc error: code = NotFound desc = could not find container \"42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7\": container with ID starting with 42035a7698665a169d7fc52d7deb86e5101815ae3500a3670b6da0983d26d9f7 not found: ID does not exist"
Sep 17 15:55:41.891161 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.891140    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"]
Sep 17 15:55:41.897606 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:41.897586    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-5kzf7"]
Sep 17 15:55:43.493996 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:43.493960    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 15:55:43.557020 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:43.556983    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" path="/var/lib/kubelet/pods/cb3faea3-47f6-4be2-9cc7-607c84e60e3d/volumes"
Sep 17 15:55:46.896401 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:46.896371    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerID="d0b9da547738ad1e397ae75eb73d93def3c94f526d555cd0728d744477b12233" exitCode=0
Sep 17 15:55:46.896861 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:46.896441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerDied","Data":"d0b9da547738ad1e397ae75eb73d93def3c94f526d555cd0728d744477b12233"}
Sep 17 15:55:48.494499 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:48.494437    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 15:55:48.498795 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:48.498767    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:55:48.500333 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:48.500308    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:55:53.494106 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:53.494063    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 15:55:53.494524 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:53.494188    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:55:58.494037 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:58.493998    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 15:55:58.498363 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:58.498328    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:55:58.499921 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:55:58.499891    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:03.494105 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:03.494054    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 15:56:08.494247 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:08.494207    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Sep 17 15:56:08.498585 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:08.498557    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Sep 17 15:56:08.498714 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:08.498686    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:56:08.500064 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:08.500039    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 15:56:08.500158 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:08.500133    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:56:11.998361 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:11.998326    2581 generic.go:356] "Generic (PLEG): container finished" podID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerID="e3eaa8fac7e7304730a0c23ed06dc89d8b5838f6028f798c8563180163123d7d" exitCode=137
Sep 17 15:56:11.998740 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:11.998399    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerDied","Data":"e3eaa8fac7e7304730a0c23ed06dc89d8b5838f6028f798c8563180163123d7d"}
Sep 17 15:56:12.026937 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.026912    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:56:12.158309 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.158219    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnxk4\" (UniqueName: \"kubernetes.io/projected/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-api-access-lnxk4\") pod \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") "
Sep 17 15:56:12.158309 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.158283    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-cabundle-cert\") pod \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") "
Sep 17 15:56:12.158309 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.158307    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad73a291-22e3-4802-91a4-a2bc9c3c111e-proxy-tls\") pod \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") "
Sep 17 15:56:12.158643 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.158363    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kserve-provision-location\") pod \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") "
Sep 17 15:56:12.158643 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.158441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-rbac-proxy-sar-config\") pod \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\" (UID: \"ad73a291-22e3-4802-91a4-a2bc9c3c111e\") "
Sep 17 15:56:12.158767 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.158740    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ad73a291-22e3-4802-91a4-a2bc9c3c111e" (UID: "ad73a291-22e3-4802-91a4-a2bc9c3c111e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:12.158829 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.158782    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ad73a291-22e3-4802-91a4-a2bc9c3c111e" (UID: "ad73a291-22e3-4802-91a4-a2bc9c3c111e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:56:12.158898 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.158869    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ad73a291-22e3-4802-91a4-a2bc9c3c111e" (UID: "ad73a291-22e3-4802-91a4-a2bc9c3c111e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:56:12.160485 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.160453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-api-access-lnxk4" (OuterVolumeSpecName: "kube-api-access-lnxk4") pod "ad73a291-22e3-4802-91a4-a2bc9c3c111e" (UID: "ad73a291-22e3-4802-91a4-a2bc9c3c111e"). InnerVolumeSpecName "kube-api-access-lnxk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:56:12.160485 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.160457    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad73a291-22e3-4802-91a4-a2bc9c3c111e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ad73a291-22e3-4802-91a4-a2bc9c3c111e" (UID: "ad73a291-22e3-4802-91a4-a2bc9c3c111e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:56:12.259533 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.259501    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:56:12.259533 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.259527    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:56:12.259533 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.259537    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnxk4\" (UniqueName: \"kubernetes.io/projected/ad73a291-22e3-4802-91a4-a2bc9c3c111e-kube-api-access-lnxk4\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:56:12.259774 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.259547    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ad73a291-22e3-4802-91a4-a2bc9c3c111e-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:56:12.259774 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:12.259556    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ad73a291-22e3-4802-91a4-a2bc9c3c111e-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:56:13.004619 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:13.004587    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn" event={"ID":"ad73a291-22e3-4802-91a4-a2bc9c3c111e","Type":"ContainerDied","Data":"72130ea46d42ddc4e88090c3e33c8a47c9e2cb930bac36b561e7cc57269456d8"}
Sep 17 15:56:13.004619 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:13.004611    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"
Sep 17 15:56:13.005077 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:13.004657    2581 scope.go:117] "RemoveContainer" containerID="e3eaa8fac7e7304730a0c23ed06dc89d8b5838f6028f798c8563180163123d7d"
Sep 17 15:56:13.012931 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:13.012915    2581 scope.go:117] "RemoveContainer" containerID="3a97bee7d119d0f4fefeb52c52957c2d8fdff2b14f47b235d4e6ff0e86c7d6c0"
Sep 17 15:56:13.019882 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:13.019864    2581 scope.go:117] "RemoveContainer" containerID="d0b9da547738ad1e397ae75eb73d93def3c94f526d555cd0728d744477b12233"
Sep 17 15:56:13.026634 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:13.026614    2581 scope.go:117] "RemoveContainer" containerID="138d0974b865608a1702c3ed28e50043a788b4090687e021e15de828fc42da94"
Sep 17 15:56:13.033666 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:13.033644    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"]
Sep 17 15:56:13.035390 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:13.035371    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-76485c66c8-92wfn"]
Sep 17 15:56:13.557174 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:13.557140    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" path="/var/lib/kubelet/pods/ad73a291-22e3-4802-91a4-a2bc9c3c111e/volumes"
Sep 17 15:56:28.269825 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.269792    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"]
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270105    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerName="kube-rbac-proxy"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270116    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerName="kube-rbac-proxy"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270126    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="storage-initializer"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270133    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="storage-initializer"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270141    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270146    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270160    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270165    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270173    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270179    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270186    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerName="kserve-container"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270192    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerName="kserve-container"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270203    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="storage-initializer"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270208    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="storage-initializer"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270216    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270221    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270228    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270233    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent"
Sep 17 15:56:28.270232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270239    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy"
Sep 17 15:56:28.270847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270244    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy"
Sep 17 15:56:28.270847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270291    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kserve-container"
Sep 17 15:56:28.270847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270298    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kube-rbac-proxy"
Sep 17 15:56:28.270847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270304    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="agent"
Sep 17 15:56:28.270847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270309    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerName="kube-rbac-proxy"
Sep 17 15:56:28.270847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270315    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4907dc-28be-458c-b0c9-6930562a35c3" containerName="kserve-container"
Sep 17 15:56:28.270847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270322    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="kube-rbac-proxy"
Sep 17 15:56:28.270847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270328    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad73a291-22e3-4802-91a4-a2bc9c3c111e" containerName="agent"
Sep 17 15:56:28.270847 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.270334    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb3faea3-47f6-4be2-9cc7-607c84e60e3d" containerName="kserve-container"
Sep 17 15:56:28.273428 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.273412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.276546 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.276523    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 15:56:28.276546 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.276523    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 17 15:56:28.276808 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.276792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 15:56:28.276879 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.276821    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 15:56:28.277043 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.277025    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 15:56:28.277110 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.277079    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 15:56:28.277433 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.277418    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 15:56:28.288265 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.288243    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"]
Sep 17 15:56:28.394003 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.393975    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whtjv\" (UniqueName: \"kubernetes.io/projected/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-api-access-whtjv\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.394126 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.394025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.394126 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.394046    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7adbbc85-700a-4e16-8753-2b019f16ac6e-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.394126 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.394070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.394126 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.394094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7adbbc85-700a-4e16-8753-2b019f16ac6e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.495345 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.495315    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whtjv\" (UniqueName: \"kubernetes.io/projected/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-api-access-whtjv\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.495542 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.495363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.495542 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.495389    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7adbbc85-700a-4e16-8753-2b019f16ac6e-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.495542 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.495409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.495542 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.495431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7adbbc85-700a-4e16-8753-2b019f16ac6e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.495859 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.495825    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7adbbc85-700a-4e16-8753-2b019f16ac6e-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.496118 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.496084    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.496213 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.496146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.497895 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.497870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7adbbc85-700a-4e16-8753-2b019f16ac6e-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.505007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.504979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whtjv\" (UniqueName: \"kubernetes.io/projected/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-api-access-whtjv\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.582810 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.582778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:28.865807 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:28.865735    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"]
Sep 17 15:56:28.868956 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:56:28.868926    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adbbc85_700a_4e16_8753_2b019f16ac6e.slice/crio-d48f14d7fcbaf864aa6b403e3bad598445b09372fe142b20f3379a59079d9b23 WatchSource:0}: Error finding container d48f14d7fcbaf864aa6b403e3bad598445b09372fe142b20f3379a59079d9b23: Status 404 returned error can't find the container with id d48f14d7fcbaf864aa6b403e3bad598445b09372fe142b20f3379a59079d9b23
Sep 17 15:56:29.074846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:29.074815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" event={"ID":"7adbbc85-700a-4e16-8753-2b019f16ac6e","Type":"ContainerStarted","Data":"f9b96e032cb1734329691aa81b56a68b1aecbf98ce325a6c2bd3eb818aad3183"}
Sep 17 15:56:29.075023 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:29.074868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" event={"ID":"7adbbc85-700a-4e16-8753-2b019f16ac6e","Type":"ContainerStarted","Data":"d48f14d7fcbaf864aa6b403e3bad598445b09372fe142b20f3379a59079d9b23"}
Sep 17 15:56:33.095181 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:33.095149    2581 generic.go:356] "Generic (PLEG): container finished" podID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerID="f9b96e032cb1734329691aa81b56a68b1aecbf98ce325a6c2bd3eb818aad3183" exitCode=0
Sep 17 15:56:33.095617 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:33.095219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" event={"ID":"7adbbc85-700a-4e16-8753-2b019f16ac6e","Type":"ContainerDied","Data":"f9b96e032cb1734329691aa81b56a68b1aecbf98ce325a6c2bd3eb818aad3183"}
Sep 17 15:56:34.100938 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:34.100905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" event={"ID":"7adbbc85-700a-4e16-8753-2b019f16ac6e","Type":"ContainerStarted","Data":"514f30b23ca7e7220ba19d51b69a5fd4d18aab0aabeae1c9596d74c3767f2b01"}
Sep 17 15:56:34.101318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:34.100966    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" event={"ID":"7adbbc85-700a-4e16-8753-2b019f16ac6e","Type":"ContainerStarted","Data":"a747151c6aed3fd994a28cb2c59733a721017d8ba1919eb0cb4c0ee3168a7ca1"}
Sep 17 15:56:34.101318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:34.101146    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:34.125588 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:34.125540    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" podStartSLOduration=2.289596185 podStartE2EDuration="6.125524848s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.835928663s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:56:34.123976592 +0000 UTC m=+1085.047551182" watchObservedRunningTime="2026-09-17 15:56:34.125524848 +0000 UTC m=+1085.049099415"
Sep 17 15:56:35.105548 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:35.105517    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:56:41.115093 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:56:41.115061    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:57:11.116222 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:11.116183    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:57:21.116534 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:21.116497    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:57:31.115846 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:31.115806    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:57:41.115702 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:41.115659    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:57:51.714740 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:51.714706    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:57:58.339439 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:58.339407    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"]
Sep 17 15:57:58.339957 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:58.339702    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kserve-container" containerID="cri-o://a747151c6aed3fd994a28cb2c59733a721017d8ba1919eb0cb4c0ee3168a7ca1" gracePeriod=30
Sep 17 15:57:58.339957 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:58.339758    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kube-rbac-proxy" containerID="cri-o://514f30b23ca7e7220ba19d51b69a5fd4d18aab0aabeae1c9596d74c3767f2b01" gracePeriod=30
Sep 17 15:57:59.455612 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:59.455581    2581 generic.go:356] "Generic (PLEG): container finished" podID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerID="514f30b23ca7e7220ba19d51b69a5fd4d18aab0aabeae1c9596d74c3767f2b01" exitCode=2
Sep 17 15:57:59.455972 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:57:59.455653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" event={"ID":"7adbbc85-700a-4e16-8753-2b019f16ac6e","Type":"ContainerDied","Data":"514f30b23ca7e7220ba19d51b69a5fd4d18aab0aabeae1c9596d74c3767f2b01"}
Sep 17 15:58:01.110185 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:01.110142    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Sep 17 15:58:01.115635 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:01.115600    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.30:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.30:8080: connect: connection refused"
Sep 17 15:58:03.474315 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.474287    2581 generic.go:356] "Generic (PLEG): container finished" podID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerID="a747151c6aed3fd994a28cb2c59733a721017d8ba1919eb0cb4c0ee3168a7ca1" exitCode=0
Sep 17 15:58:03.474704 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.474362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" event={"ID":"7adbbc85-700a-4e16-8753-2b019f16ac6e","Type":"ContainerDied","Data":"a747151c6aed3fd994a28cb2c59733a721017d8ba1919eb0cb4c0ee3168a7ca1"}
Sep 17 15:58:03.474704 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.474457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79" event={"ID":"7adbbc85-700a-4e16-8753-2b019f16ac6e","Type":"ContainerDied","Data":"d48f14d7fcbaf864aa6b403e3bad598445b09372fe142b20f3379a59079d9b23"}
Sep 17 15:58:03.474704 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.474497    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d48f14d7fcbaf864aa6b403e3bad598445b09372fe142b20f3379a59079d9b23"
Sep 17 15:58:03.478610 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.478594    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:58:03.578714 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.578631    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-cabundle-cert\") pod \"7adbbc85-700a-4e16-8753-2b019f16ac6e\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") "
Sep 17 15:58:03.578714 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.578679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whtjv\" (UniqueName: \"kubernetes.io/projected/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-api-access-whtjv\") pod \"7adbbc85-700a-4e16-8753-2b019f16ac6e\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") "
Sep 17 15:58:03.578933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.578732    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7adbbc85-700a-4e16-8753-2b019f16ac6e-kserve-provision-location\") pod \"7adbbc85-700a-4e16-8753-2b019f16ac6e\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") "
Sep 17 15:58:03.578933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.578756    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7adbbc85-700a-4e16-8753-2b019f16ac6e-proxy-tls\") pod \"7adbbc85-700a-4e16-8753-2b019f16ac6e\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") "
Sep 17 15:58:03.578933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.578801    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-rbac-proxy-sar-config\") pod \"7adbbc85-700a-4e16-8753-2b019f16ac6e\" (UID: \"7adbbc85-700a-4e16-8753-2b019f16ac6e\") "
Sep 17 15:58:03.579113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.578980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7adbbc85-700a-4e16-8753-2b019f16ac6e" (UID: "7adbbc85-700a-4e16-8753-2b019f16ac6e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:03.579113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.579066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7adbbc85-700a-4e16-8753-2b019f16ac6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7adbbc85-700a-4e16-8753-2b019f16ac6e" (UID: "7adbbc85-700a-4e16-8753-2b019f16ac6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:58:03.579113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.579084    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:58:03.579267 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.579153    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7adbbc85-700a-4e16-8753-2b019f16ac6e" (UID: "7adbbc85-700a-4e16-8753-2b019f16ac6e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:58:03.580811 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.580788    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7adbbc85-700a-4e16-8753-2b019f16ac6e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7adbbc85-700a-4e16-8753-2b019f16ac6e" (UID: "7adbbc85-700a-4e16-8753-2b019f16ac6e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:58:03.580899 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.580823    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-api-access-whtjv" (OuterVolumeSpecName: "kube-api-access-whtjv") pod "7adbbc85-700a-4e16-8753-2b019f16ac6e" (UID: "7adbbc85-700a-4e16-8753-2b019f16ac6e"). InnerVolumeSpecName "kube-api-access-whtjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:58:03.664488 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.664437    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"]
Sep 17 15:58:03.664757 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.664745    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kserve-container"
Sep 17 15:58:03.664801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.664759    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kserve-container"
Sep 17 15:58:03.664801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.664775    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="storage-initializer"
Sep 17 15:58:03.664801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.664780    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="storage-initializer"
Sep 17 15:58:03.664801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.664795    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kube-rbac-proxy"
Sep 17 15:58:03.664801 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.664802    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kube-rbac-proxy"
Sep 17 15:58:03.664954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.664849    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kserve-container"
Sep 17 15:58:03.664954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.664859    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" containerName="kube-rbac-proxy"
Sep 17 15:58:03.667980 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.667964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.672615 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.672598    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 17 15:58:03.673003 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.672978    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 15:58:03.679499 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.679431    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7adbbc85-700a-4e16-8753-2b019f16ac6e-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:58:03.679499 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.679451    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7adbbc85-700a-4e16-8753-2b019f16ac6e-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:58:03.679499 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.679476    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:58:03.679499 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.679486    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whtjv\" (UniqueName: \"kubernetes.io/projected/7adbbc85-700a-4e16-8753-2b019f16ac6e-kube-api-access-whtjv\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:58:03.681318 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.681293    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"]
Sep 17 15:58:03.780232 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.780198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l87bq\" (UniqueName: \"kubernetes.io/projected/3745dddd-468f-4a25-9525-be689fef7d34-kube-api-access-l87bq\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.780393 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.780242    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.780393 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.780337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.780528 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.780397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3745dddd-468f-4a25-9525-be689fef7d34-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.780528 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.780416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3745dddd-468f-4a25-9525-be689fef7d34-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.881290 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.881205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3745dddd-468f-4a25-9525-be689fef7d34-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.881290 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.881241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3745dddd-468f-4a25-9525-be689fef7d34-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.881290 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.881266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l87bq\" (UniqueName: \"kubernetes.io/projected/3745dddd-468f-4a25-9525-be689fef7d34-kube-api-access-l87bq\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.881290 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.881285    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.881675 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.881331    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.881738 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.881716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3745dddd-468f-4a25-9525-be689fef7d34-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.882120 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.882100    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.882167 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.882102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.883752 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.883733    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3745dddd-468f-4a25-9525-be689fef7d34-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.890517 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.890492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l87bq\" (UniqueName: \"kubernetes.io/projected/3745dddd-468f-4a25-9525-be689fef7d34-kube-api-access-l87bq\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:03.978123 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:03.978090    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:04.253276 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:04.253135    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"]
Sep 17 15:58:04.255446 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:58:04.255416    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3745dddd_468f_4a25_9525_be689fef7d34.slice/crio-60b6877bc7e780e38b48e0bb3b7dfd0f80fa4e97f3d303330ea2bbda94d5f0b7 WatchSource:0}: Error finding container 60b6877bc7e780e38b48e0bb3b7dfd0f80fa4e97f3d303330ea2bbda94d5f0b7: Status 404 returned error can't find the container with id 60b6877bc7e780e38b48e0bb3b7dfd0f80fa4e97f3d303330ea2bbda94d5f0b7
Sep 17 15:58:04.479954 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:04.479917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" event={"ID":"3745dddd-468f-4a25-9525-be689fef7d34","Type":"ContainerStarted","Data":"ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615"}
Sep 17 15:58:04.480430 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:04.479964    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"
Sep 17 15:58:04.480430 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:04.479975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" event={"ID":"3745dddd-468f-4a25-9525-be689fef7d34","Type":"ContainerStarted","Data":"60b6877bc7e780e38b48e0bb3b7dfd0f80fa4e97f3d303330ea2bbda94d5f0b7"}
Sep 17 15:58:04.522300 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:04.522217    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"]
Sep 17 15:58:04.526479 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:04.526439    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-rmp79"]
Sep 17 15:58:05.562481 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:05.562443    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7adbbc85-700a-4e16-8753-2b019f16ac6e" path="/var/lib/kubelet/pods/7adbbc85-700a-4e16-8753-2b019f16ac6e/volumes"
Sep 17 15:58:08.497906 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:08.497821    2581 generic.go:356] "Generic (PLEG): container finished" podID="3745dddd-468f-4a25-9525-be689fef7d34" containerID="ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615" exitCode=0
Sep 17 15:58:08.497906 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:08.497895    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" event={"ID":"3745dddd-468f-4a25-9525-be689fef7d34","Type":"ContainerDied","Data":"ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615"}
Sep 17 15:58:09.504621 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:09.504588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" event={"ID":"3745dddd-468f-4a25-9525-be689fef7d34","Type":"ContainerStarted","Data":"fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882"}
Sep 17 15:58:09.505039 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:09.504650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" event={"ID":"3745dddd-468f-4a25-9525-be689fef7d34","Type":"ContainerStarted","Data":"291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58"}
Sep 17 15:58:09.505039 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:09.504950    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:09.505039 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:09.504992    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:09.540310 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:09.540257    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podStartSLOduration=2.7333979299999998 podStartE2EDuration="6.540240909s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.806842979s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:58:09.53807502 +0000 UTC m=+1180.461649602" watchObservedRunningTime="2026-09-17 15:58:09.540240909 +0000 UTC m=+1180.463815475"
Sep 17 15:58:15.513792 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:15.513763    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:58:45.514449 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:45.514396    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:58:55.514349 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:58:55.514311    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:59:05.515162 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:05.515122    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:59:15.515126 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:15.515090    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:59:17.552821 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:17.552778    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:59:28.114533 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:28.114495    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:59:33.596620 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:33.596588    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"]
Sep 17 15:59:33.597029 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:33.596897    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container" containerID="cri-o://291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58" gracePeriod=30
Sep 17 15:59:33.597029 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:33.596948    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kube-rbac-proxy" containerID="cri-o://fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882" gracePeriod=30
Sep 17 15:59:33.855711 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:33.855633    2581 generic.go:356] "Generic (PLEG): container finished" podID="3745dddd-468f-4a25-9525-be689fef7d34" containerID="fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882" exitCode=2
Sep 17 15:59:33.855711 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:33.855693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" event={"ID":"3745dddd-468f-4a25-9525-be689fef7d34","Type":"ContainerDied","Data":"fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882"}
Sep 17 15:59:35.509785 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:35.509745    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Sep 17 15:59:37.553021 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:37.552981    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.31:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.31:8080: connect: connection refused"
Sep 17 15:59:38.436263 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.436241    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:59:38.577068 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577041    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3745dddd-468f-4a25-9525-be689fef7d34-proxy-tls\") pod \"3745dddd-468f-4a25-9525-be689fef7d34\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") "
Sep 17 15:59:38.577485 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577110    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-cabundle-cert\") pod \"3745dddd-468f-4a25-9525-be689fef7d34\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") "
Sep 17 15:59:38.577485 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l87bq\" (UniqueName: \"kubernetes.io/projected/3745dddd-468f-4a25-9525-be689fef7d34-kube-api-access-l87bq\") pod \"3745dddd-468f-4a25-9525-be689fef7d34\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") "
Sep 17 15:59:38.577485 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577151    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-kube-rbac-proxy-sar-config\") pod \"3745dddd-468f-4a25-9525-be689fef7d34\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") "
Sep 17 15:59:38.577485 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577209    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3745dddd-468f-4a25-9525-be689fef7d34-kserve-provision-location\") pod \"3745dddd-468f-4a25-9525-be689fef7d34\" (UID: \"3745dddd-468f-4a25-9525-be689fef7d34\") "
Sep 17 15:59:38.577723 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577493    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3745dddd-468f-4a25-9525-be689fef7d34" (UID: "3745dddd-468f-4a25-9525-be689fef7d34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:38.577723 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577549    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3745dddd-468f-4a25-9525-be689fef7d34" (UID: "3745dddd-468f-4a25-9525-be689fef7d34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 15:59:38.577723 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577701    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3745dddd-468f-4a25-9525-be689fef7d34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3745dddd-468f-4a25-9525-be689fef7d34" (UID: "3745dddd-468f-4a25-9525-be689fef7d34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 15:59:38.577887 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577831    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:59:38.577887 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577849    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3745dddd-468f-4a25-9525-be689fef7d34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:59:38.577887 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.577864    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3745dddd-468f-4a25-9525-be689fef7d34-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:59:38.579215 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.579191    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3745dddd-468f-4a25-9525-be689fef7d34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3745dddd-468f-4a25-9525-be689fef7d34" (UID: "3745dddd-468f-4a25-9525-be689fef7d34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 15:59:38.579313 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.579242    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3745dddd-468f-4a25-9525-be689fef7d34-kube-api-access-l87bq" (OuterVolumeSpecName: "kube-api-access-l87bq") pod "3745dddd-468f-4a25-9525-be689fef7d34" (UID: "3745dddd-468f-4a25-9525-be689fef7d34"). InnerVolumeSpecName "kube-api-access-l87bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 15:59:38.678237 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.678205    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3745dddd-468f-4a25-9525-be689fef7d34-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:59:38.678237 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.678233    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l87bq\" (UniqueName: \"kubernetes.io/projected/3745dddd-468f-4a25-9525-be689fef7d34-kube-api-access-l87bq\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 15:59:38.826384 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.826352    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"]
Sep 17 15:59:38.826690 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.826677    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="storage-initializer"
Sep 17 15:59:38.826750 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.826691    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="storage-initializer"
Sep 17 15:59:38.826750 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.826703    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container"
Sep 17 15:59:38.826750 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.826709    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container"
Sep 17 15:59:38.826750 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.826725    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kube-rbac-proxy"
Sep 17 15:59:38.826750 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.826730    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kube-rbac-proxy"
Sep 17 15:59:38.826951 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.826785    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kube-rbac-proxy"
Sep 17 15:59:38.826951 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.826793    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3745dddd-468f-4a25-9525-be689fef7d34" containerName="kserve-container"
Sep 17 15:59:38.829582 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.829540    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:38.834896 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:38.834867    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:38.835068 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:38.835044    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-predictive-lightgbm-v2-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\"" type="*v1.Secret"
Sep 17 15:59:38.835399 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:38.835379    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 17 15:59:38.852009 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.851988    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"]
Sep 17 15:59:38.878007 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.877984    2581 generic.go:356] "Generic (PLEG): container finished" podID="3745dddd-468f-4a25-9525-be689fef7d34" containerID="291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58" exitCode=0
Sep 17 15:59:38.878097 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.878032    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"
Sep 17 15:59:38.878097 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.878061    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" event={"ID":"3745dddd-468f-4a25-9525-be689fef7d34","Type":"ContainerDied","Data":"291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58"}
Sep 17 15:59:38.878174 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.878120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2" event={"ID":"3745dddd-468f-4a25-9525-be689fef7d34","Type":"ContainerDied","Data":"60b6877bc7e780e38b48e0bb3b7dfd0f80fa4e97f3d303330ea2bbda94d5f0b7"}
Sep 17 15:59:38.878174 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.878150    2581 scope.go:117] "RemoveContainer" containerID="fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882"
Sep 17 15:59:38.886590 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.886573    2581 scope.go:117] "RemoveContainer" containerID="291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58"
Sep 17 15:59:38.895655 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.895635    2581 scope.go:117] "RemoveContainer" containerID="ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615"
Sep 17 15:59:38.902966 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.902950    2581 scope.go:117] "RemoveContainer" containerID="fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882"
Sep 17 15:59:38.903231 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:38.903214    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882\": container with ID starting with fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882 not found: ID does not exist" containerID="fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882"
Sep 17 15:59:38.903277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.903239    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882"} err="failed to get container status \"fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882\": rpc error: code = NotFound desc = could not find container \"fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882\": container with ID starting with fabf629e4f01e0a7c20872394936f6bcab39a392890aeaa31361f373f9fe7882 not found: ID does not exist"
Sep 17 15:59:38.903277 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.903255    2581 scope.go:117] "RemoveContainer" containerID="291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58"
Sep 17 15:59:38.903488 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:38.903455    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58\": container with ID starting with 291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58 not found: ID does not exist" containerID="291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58"
Sep 17 15:59:38.903551 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.903495    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58"} err="failed to get container status \"291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58\": rpc error: code = NotFound desc = could not find container \"291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58\": container with ID starting with 291db4b51fa5a1aaf662bbf051b49fd59105da0a6bf7ef685a3225f41eff1e58 not found: ID does not exist"
Sep 17 15:59:38.903551 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.903517    2581 scope.go:117] "RemoveContainer" containerID="ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615"
Sep 17 15:59:38.903870 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:38.903842    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615\": container with ID starting with ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615 not found: ID does not exist" containerID="ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615"
Sep 17 15:59:38.903933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.903869    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615"} err="failed to get container status \"ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615\": rpc error: code = NotFound desc = could not find container \"ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615\": container with ID starting with ba5d6561558d76f2e3113a31373b739af142dc626fa9c1b29777e9275d93c615 not found: ID does not exist"
Sep 17 15:59:38.913982 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.913958    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"]
Sep 17 15:59:38.919449 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.919431    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-rdsk2"]
Sep 17 15:59:38.980742 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.980711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:38.980849 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.980750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:38.980849 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.980791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd2c367-40d7-4cb8-ab04-007a943af53c-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:38.980933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.980843    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwvgk\" (UniqueName: \"kubernetes.io/projected/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-api-access-fwvgk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:38.980933 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:38.980886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd2c367-40d7-4cb8-ab04-007a943af53c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:39.082152 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:39.082082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:39.082152 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:39.082136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd2c367-40d7-4cb8-ab04-007a943af53c-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:39.082360 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:39.082176    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwvgk\" (UniqueName: \"kubernetes.io/projected/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-api-access-fwvgk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:39.082360 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:39.082219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd2c367-40d7-4cb8-ab04-007a943af53c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:39.082360 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:39.082345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:39.082591 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:39.082574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd2c367-40d7-4cb8-ab04-007a943af53c-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:39.082832 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:39.082812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:39.090896 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:39.090877    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwvgk\" (UniqueName: \"kubernetes.io/projected/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-api-access-fwvgk\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:39.557103 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:39.557074    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3745dddd-468f-4a25-9525-be689fef7d34" path="/var/lib/kubelet/pods/3745dddd-468f-4a25-9525-be689fef7d34/volumes"
Sep 17 15:59:40.083432 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:40.083399    2581 configmap.go:193] Couldn't get configMap kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config: failed to sync configmap cache: timed out waiting for the condition
Sep 17 15:59:40.083836 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:40.083406    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: failed to sync secret cache: timed out waiting for the condition
Sep 17 15:59:40.083836 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:40.083499    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-rbac-proxy-sar-config podName:9cd2c367-40d7-4cb8-ab04-007a943af53c nodeName:}" failed. No retries permitted until 2026-09-17 15:59:40.583480594 +0000 UTC m=+1271.507055152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-sar-config" (UniqueName: "kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-rbac-proxy-sar-config") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" (UID: "9cd2c367-40d7-4cb8-ab04-007a943af53c") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 15:59:40.083836 ip-10-0-140-157 kubenswrapper[2581]: E0917 15:59:40.083561    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd2c367-40d7-4cb8-ab04-007a943af53c-proxy-tls podName:9cd2c367-40d7-4cb8-ab04-007a943af53c nodeName:}" failed. No retries permitted until 2026-09-17 15:59:40.583547086 +0000 UTC m=+1271.507121630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9cd2c367-40d7-4cb8-ab04-007a943af53c-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" (UID: "9cd2c367-40d7-4cb8-ab04-007a943af53c") : failed to sync secret cache: timed out waiting for the condition
Sep 17 15:59:40.143228 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:40.143199    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 17 15:59:40.397051 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:40.396977    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 17 15:59:40.593512 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:40.593477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd2c367-40d7-4cb8-ab04-007a943af53c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:40.593512 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:40.593516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:40.594078 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:40.594057    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:40.595899 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:40.595881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd2c367-40d7-4cb8-ab04-007a943af53c-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:40.639429 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:40.639393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:40.913621 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:40.913599    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"]
Sep 17 15:59:40.915941 ip-10-0-140-157 kubenswrapper[2581]: W0917 15:59:40.915916    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd2c367_40d7_4cb8_ab04_007a943af53c.slice/crio-1c1c236cf0d765ebfb8d75b257d3291076cc8581a477e92bcf4fead1679d3b6d WatchSource:0}: Error finding container 1c1c236cf0d765ebfb8d75b257d3291076cc8581a477e92bcf4fead1679d3b6d: Status 404 returned error can't find the container with id 1c1c236cf0d765ebfb8d75b257d3291076cc8581a477e92bcf4fead1679d3b6d
Sep 17 15:59:41.892670 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:41.892635    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" event={"ID":"9cd2c367-40d7-4cb8-ab04-007a943af53c","Type":"ContainerStarted","Data":"4feed61a911bf5d512a1ba410e7ea7f0b48afdd0e73fd0cff49cb2e669efd561"}
Sep 17 15:59:41.893063 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:41.892696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" event={"ID":"9cd2c367-40d7-4cb8-ab04-007a943af53c","Type":"ContainerStarted","Data":"1c1c236cf0d765ebfb8d75b257d3291076cc8581a477e92bcf4fead1679d3b6d"}
Sep 17 15:59:45.911807 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:45.911773    2581 generic.go:356] "Generic (PLEG): container finished" podID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerID="4feed61a911bf5d512a1ba410e7ea7f0b48afdd0e73fd0cff49cb2e669efd561" exitCode=0
Sep 17 15:59:45.912181 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:45.911841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" event={"ID":"9cd2c367-40d7-4cb8-ab04-007a943af53c","Type":"ContainerDied","Data":"4feed61a911bf5d512a1ba410e7ea7f0b48afdd0e73fd0cff49cb2e669efd561"}
Sep 17 15:59:46.922241 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:46.922205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" event={"ID":"9cd2c367-40d7-4cb8-ab04-007a943af53c","Type":"ContainerStarted","Data":"0c2790d543e28336ca736af02276003fefbe3cc6a577bd3201e198365dacbed6"}
Sep 17 15:59:46.922676 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:46.922270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" event={"ID":"9cd2c367-40d7-4cb8-ab04-007a943af53c","Type":"ContainerStarted","Data":"ebb25d249b3f4a6cacbcca88771f51278fe11efb68f8067c3a1ee0f6c63a19d6"}
Sep 17 15:59:46.922676 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:46.922482    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:46.953113 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:46.953066    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podStartSLOduration=4.967372071 podStartE2EDuration="8.953052897s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.985680826s" isStatefulPod=false podCreationTimestamp="2026-09-17 15:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 15:59:46.952639394 +0000 UTC m=+1277.876213961" watchObservedRunningTime="2026-09-17 15:59:46.953052897 +0000 UTC m=+1277.876627461"
Sep 17 15:59:47.931204 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:47.931167    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 15:59:53.940479 ip-10-0-140-157 kubenswrapper[2581]: I0917 15:59:53.940440    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 16:00:23.941443 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:00:23.941405    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:00:33.941760 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:00:33.941713    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:00:43.941549 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:00:43.941499    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:00:53.941177 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:00:53.941093    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:01:04.514983 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:04.514946    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 16:01:08.909750 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:08.909718    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"]
Sep 17 16:01:08.910125 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:08.909975    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kube-rbac-proxy" containerID="cri-o://0c2790d543e28336ca736af02276003fefbe3cc6a577bd3201e198365dacbed6" gracePeriod=30
Sep 17 16:01:08.910125 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:08.909952    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kserve-container" containerID="cri-o://ebb25d249b3f4a6cacbcca88771f51278fe11efb68f8067c3a1ee0f6c63a19d6" gracePeriod=30
Sep 17 16:01:08.935875 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:08.935837    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 16:01:09.274646 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:09.274561    2581 generic.go:356] "Generic (PLEG): container finished" podID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerID="0c2790d543e28336ca736af02276003fefbe3cc6a577bd3201e198365dacbed6" exitCode=2
Sep 17 16:01:09.274646 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:09.274636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" event={"ID":"9cd2c367-40d7-4cb8-ab04-007a943af53c","Type":"ContainerDied","Data":"0c2790d543e28336ca736af02276003fefbe3cc6a577bd3201e198365dacbed6"}
Sep 17 16:01:13.936329 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:13.936292    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Sep 17 16:01:13.941912 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:13.941881    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.32:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.32:8080: connect: connection refused"
Sep 17 16:01:14.298437 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.298410    2581 generic.go:356] "Generic (PLEG): container finished" podID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerID="ebb25d249b3f4a6cacbcca88771f51278fe11efb68f8067c3a1ee0f6c63a19d6" exitCode=0
Sep 17 16:01:14.298576 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.298453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" event={"ID":"9cd2c367-40d7-4cb8-ab04-007a943af53c","Type":"ContainerDied","Data":"ebb25d249b3f4a6cacbcca88771f51278fe11efb68f8067c3a1ee0f6c63a19d6"}
Sep 17 16:01:14.352149 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.352125    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 16:01:14.459330 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.459300    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd2c367-40d7-4cb8-ab04-007a943af53c-proxy-tls\") pod \"9cd2c367-40d7-4cb8-ab04-007a943af53c\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") "
Sep 17 16:01:14.459488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.459340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-cabundle-cert\") pod \"9cd2c367-40d7-4cb8-ab04-007a943af53c\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") "
Sep 17 16:01:14.459488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.459359    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fwvgk\" (UniqueName: \"kubernetes.io/projected/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-api-access-fwvgk\") pod \"9cd2c367-40d7-4cb8-ab04-007a943af53c\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") "
Sep 17 16:01:14.459582 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.459541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-rbac-proxy-sar-config\") pod \"9cd2c367-40d7-4cb8-ab04-007a943af53c\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") "
Sep 17 16:01:14.459627 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.459610    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd2c367-40d7-4cb8-ab04-007a943af53c-kserve-provision-location\") pod \"9cd2c367-40d7-4cb8-ab04-007a943af53c\" (UID: \"9cd2c367-40d7-4cb8-ab04-007a943af53c\") "
Sep 17 16:01:14.459679 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.459654    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9cd2c367-40d7-4cb8-ab04-007a943af53c" (UID: "9cd2c367-40d7-4cb8-ab04-007a943af53c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:14.459860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.459845    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:01:14.459932 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.459875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9cd2c367-40d7-4cb8-ab04-007a943af53c" (UID: "9cd2c367-40d7-4cb8-ab04-007a943af53c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:01:14.459932 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.459912    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cd2c367-40d7-4cb8-ab04-007a943af53c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9cd2c367-40d7-4cb8-ab04-007a943af53c" (UID: "9cd2c367-40d7-4cb8-ab04-007a943af53c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:01:14.461528 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.461508    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd2c367-40d7-4cb8-ab04-007a943af53c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9cd2c367-40d7-4cb8-ab04-007a943af53c" (UID: "9cd2c367-40d7-4cb8-ab04-007a943af53c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:01:14.461614 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.461505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-api-access-fwvgk" (OuterVolumeSpecName: "kube-api-access-fwvgk") pod "9cd2c367-40d7-4cb8-ab04-007a943af53c" (UID: "9cd2c367-40d7-4cb8-ab04-007a943af53c"). InnerVolumeSpecName "kube-api-access-fwvgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:14.560438 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.560413    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:01:14.560438 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.560436    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9cd2c367-40d7-4cb8-ab04-007a943af53c-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:01:14.560648 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.560446    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9cd2c367-40d7-4cb8-ab04-007a943af53c-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:01:14.560648 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:14.560455    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwvgk\" (UniqueName: \"kubernetes.io/projected/9cd2c367-40d7-4cb8-ab04-007a943af53c-kube-api-access-fwvgk\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:01:15.304476 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:15.304447    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"
Sep 17 16:01:15.304885 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:15.304442    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j" event={"ID":"9cd2c367-40d7-4cb8-ab04-007a943af53c","Type":"ContainerDied","Data":"1c1c236cf0d765ebfb8d75b257d3291076cc8581a477e92bcf4fead1679d3b6d"}
Sep 17 16:01:15.304885 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:15.304606    2581 scope.go:117] "RemoveContainer" containerID="0c2790d543e28336ca736af02276003fefbe3cc6a577bd3201e198365dacbed6"
Sep 17 16:01:15.312879 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:15.312862    2581 scope.go:117] "RemoveContainer" containerID="ebb25d249b3f4a6cacbcca88771f51278fe11efb68f8067c3a1ee0f6c63a19d6"
Sep 17 16:01:15.320402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:15.320381    2581 scope.go:117] "RemoveContainer" containerID="4feed61a911bf5d512a1ba410e7ea7f0b48afdd0e73fd0cff49cb2e669efd561"
Sep 17 16:01:15.328661 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:15.328643    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"]
Sep 17 16:01:15.334541 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:15.334521    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-jqd2j"]
Sep 17 16:01:15.557387 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:15.557318    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" path="/var/lib/kubelet/pods/9cd2c367-40d7-4cb8-ab04-007a943af53c/volumes"
Sep 17 16:01:31.758799 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.758767    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-345fb311"]
Sep 17 16:01:31.759139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.759078    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kserve-container"
Sep 17 16:01:31.759139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.759089    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kserve-container"
Sep 17 16:01:31.759139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.759097    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kube-rbac-proxy"
Sep 17 16:01:31.759139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.759103    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kube-rbac-proxy"
Sep 17 16:01:31.759139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.759111    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="storage-initializer"
Sep 17 16:01:31.759139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.759117    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="storage-initializer"
Sep 17 16:01:31.759338 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.759163    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kserve-container"
Sep 17 16:01:31.759338 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.759176    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cd2c367-40d7-4cb8-ab04-007a943af53c" containerName="kube-rbac-proxy"
Sep 17 16:01:31.763167 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.763145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-345fb311"
Sep 17 16:01:31.767057 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.767036    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dl92p\""
Sep 17 16:01:31.767328 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.767307    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:01:31.767589 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.767562    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:01:31.768420 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.768396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-345fb311"]
Sep 17 16:01:31.893356 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.893320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-996ps\" (UniqueName: \"kubernetes.io/projected/85608f4c-ea1b-4049-a062-30aa292cd92c-kube-api-access-996ps\") pod \"curl-metrics-345fb311\" (UID: \"85608f4c-ea1b-4049-a062-30aa292cd92c\") " pod="kserve/curl-metrics-345fb311"
Sep 17 16:01:31.994045 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:31.994008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-996ps\" (UniqueName: \"kubernetes.io/projected/85608f4c-ea1b-4049-a062-30aa292cd92c-kube-api-access-996ps\") pod \"curl-metrics-345fb311\" (UID: \"85608f4c-ea1b-4049-a062-30aa292cd92c\") " pod="kserve/curl-metrics-345fb311"
Sep 17 16:01:32.004418 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:32.004390    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-996ps\" (UniqueName: \"kubernetes.io/projected/85608f4c-ea1b-4049-a062-30aa292cd92c-kube-api-access-996ps\") pod \"curl-metrics-345fb311\" (UID: \"85608f4c-ea1b-4049-a062-30aa292cd92c\") " pod="kserve/curl-metrics-345fb311"
Sep 17 16:01:32.088393 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:32.088369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-345fb311"
Sep 17 16:01:32.362358 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:32.362283    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-345fb311"]
Sep 17 16:01:32.367565 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:01:32.367536    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85608f4c_ea1b_4049_a062_30aa292cd92c.slice/crio-618c9ecb9c79c613a7c582350c0f913b99dd347439ac2925aeef059b1dba6225 WatchSource:0}: Error finding container 618c9ecb9c79c613a7c582350c0f913b99dd347439ac2925aeef059b1dba6225: Status 404 returned error can't find the container with id 618c9ecb9c79c613a7c582350c0f913b99dd347439ac2925aeef059b1dba6225
Sep 17 16:01:32.369948 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:32.369932    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:01:32.380859 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:32.380834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-345fb311" event={"ID":"85608f4c-ea1b-4049-a062-30aa292cd92c","Type":"ContainerStarted","Data":"618c9ecb9c79c613a7c582350c0f913b99dd347439ac2925aeef059b1dba6225"}
Sep 17 16:01:33.386339 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:33.386256    2581 generic.go:356] "Generic (PLEG): container finished" podID="85608f4c-ea1b-4049-a062-30aa292cd92c" containerID="fcdcbfa6fa98bbd83b44274c5ce966120ecd347ccd53ef23121d6f15da4753ce" exitCode=0
Sep 17 16:01:33.386339 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:33.386300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-345fb311" event={"ID":"85608f4c-ea1b-4049-a062-30aa292cd92c","Type":"ContainerDied","Data":"fcdcbfa6fa98bbd83b44274c5ce966120ecd347ccd53ef23121d6f15da4753ce"}
Sep 17 16:01:34.505891 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:34.505871    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-345fb311"
Sep 17 16:01:34.543912 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:34.543890    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-345fb311_85608f4c-ea1b-4049-a062-30aa292cd92c/curl-metrics-345fb311/0.log"
Sep 17 16:01:34.565138 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:34.565113    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-345fb311"]
Sep 17 16:01:34.567088 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:34.567068    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-345fb311"]
Sep 17 16:01:34.617030 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:34.616994    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-996ps\" (UniqueName: \"kubernetes.io/projected/85608f4c-ea1b-4049-a062-30aa292cd92c-kube-api-access-996ps\") pod \"85608f4c-ea1b-4049-a062-30aa292cd92c\" (UID: \"85608f4c-ea1b-4049-a062-30aa292cd92c\") "
Sep 17 16:01:34.618900 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:34.618882    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85608f4c-ea1b-4049-a062-30aa292cd92c-kube-api-access-996ps" (OuterVolumeSpecName: "kube-api-access-996ps") pod "85608f4c-ea1b-4049-a062-30aa292cd92c" (UID: "85608f4c-ea1b-4049-a062-30aa292cd92c"). InnerVolumeSpecName "kube-api-access-996ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:34.718432 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:34.718351    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-996ps\" (UniqueName: \"kubernetes.io/projected/85608f4c-ea1b-4049-a062-30aa292cd92c-kube-api-access-996ps\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:01:35.396831 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:35.396805    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="618c9ecb9c79c613a7c582350c0f913b99dd347439ac2925aeef059b1dba6225"
Sep 17 16:01:35.396998 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:35.396836    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-345fb311"
Sep 17 16:01:35.405950 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:01:35.405923    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-345fb311\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" podUID="85608f4c-ea1b-4049-a062-30aa292cd92c" pod="kserve/curl-metrics-345fb311"
Sep 17 16:01:35.557910 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:35.557880    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85608f4c-ea1b-4049-a062-30aa292cd92c" path="/var/lib/kubelet/pods/85608f4c-ea1b-4049-a062-30aa292cd92c/volumes"
Sep 17 16:01:39.969907 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:39.969877    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/curl-metrics-dcdff4d5"]
Sep 17 16:01:39.970262 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:39.970231    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85608f4c-ea1b-4049-a062-30aa292cd92c" containerName="curl-metrics-345fb311"
Sep 17 16:01:39.970262 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:39.970242    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85608f4c-ea1b-4049-a062-30aa292cd92c" containerName="curl-metrics-345fb311"
Sep 17 16:01:39.970339 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:39.970312    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="85608f4c-ea1b-4049-a062-30aa292cd92c" containerName="curl-metrics-345fb311"
Sep 17 16:01:39.973503 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:39.973485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-dcdff4d5"
Sep 17 16:01:39.975772 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:39.975753    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 16:01:39.976217 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:39.976201    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-dl92p\""
Sep 17 16:01:39.976303 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:39.976230    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 16:01:39.979425 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:39.979406    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-dcdff4d5"]
Sep 17 16:01:40.062201 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:40.062168    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf84t\" (UniqueName: \"kubernetes.io/projected/0afd3a3c-bf70-492b-ba79-489a2d0034ff-kube-api-access-wf84t\") pod \"curl-metrics-dcdff4d5\" (UID: \"0afd3a3c-bf70-492b-ba79-489a2d0034ff\") " pod="kserve/curl-metrics-dcdff4d5"
Sep 17 16:01:40.163456 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:40.163428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf84t\" (UniqueName: \"kubernetes.io/projected/0afd3a3c-bf70-492b-ba79-489a2d0034ff-kube-api-access-wf84t\") pod \"curl-metrics-dcdff4d5\" (UID: \"0afd3a3c-bf70-492b-ba79-489a2d0034ff\") " pod="kserve/curl-metrics-dcdff4d5"
Sep 17 16:01:40.171910 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:40.171888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf84t\" (UniqueName: \"kubernetes.io/projected/0afd3a3c-bf70-492b-ba79-489a2d0034ff-kube-api-access-wf84t\") pod \"curl-metrics-dcdff4d5\" (UID: \"0afd3a3c-bf70-492b-ba79-489a2d0034ff\") " pod="kserve/curl-metrics-dcdff4d5"
Sep 17 16:01:40.295474 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:40.295432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-dcdff4d5"
Sep 17 16:01:40.563094 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:40.562997    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/curl-metrics-dcdff4d5"]
Sep 17 16:01:40.566054 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:01:40.566025    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0afd3a3c_bf70_492b_ba79_489a2d0034ff.slice/crio-542aa4d3e8050dc35a93a01aea139ce70debcfc5717dc340fb45c330dc33f6ba WatchSource:0}: Error finding container 542aa4d3e8050dc35a93a01aea139ce70debcfc5717dc340fb45c330dc33f6ba: Status 404 returned error can't find the container with id 542aa4d3e8050dc35a93a01aea139ce70debcfc5717dc340fb45c330dc33f6ba
Sep 17 16:01:41.423004 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:41.422965    2581 generic.go:356] "Generic (PLEG): container finished" podID="0afd3a3c-bf70-492b-ba79-489a2d0034ff" containerID="ff55f560472be9f737944b96b722e89b508dd324ff1cda4712ac7749d947c004" exitCode=0
Sep 17 16:01:41.423365 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:41.423048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-dcdff4d5" event={"ID":"0afd3a3c-bf70-492b-ba79-489a2d0034ff","Type":"ContainerDied","Data":"ff55f560472be9f737944b96b722e89b508dd324ff1cda4712ac7749d947c004"}
Sep 17 16:01:41.423365 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:41.423089    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/curl-metrics-dcdff4d5" event={"ID":"0afd3a3c-bf70-492b-ba79-489a2d0034ff","Type":"ContainerStarted","Data":"542aa4d3e8050dc35a93a01aea139ce70debcfc5717dc340fb45c330dc33f6ba"}
Sep 17 16:01:42.546271 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:42.546249    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-dcdff4d5"
Sep 17 16:01:42.567271 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:42.567251    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_curl-metrics-dcdff4d5_0afd3a3c-bf70-492b-ba79-489a2d0034ff/curl-metrics-dcdff4d5/0.log"
Sep 17 16:01:42.590416 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:42.590394    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/curl-metrics-dcdff4d5"]
Sep 17 16:01:42.593842 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:42.593824    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/curl-metrics-dcdff4d5"]
Sep 17 16:01:42.685068 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:42.684997    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wf84t\" (UniqueName: \"kubernetes.io/projected/0afd3a3c-bf70-492b-ba79-489a2d0034ff-kube-api-access-wf84t\") pod \"0afd3a3c-bf70-492b-ba79-489a2d0034ff\" (UID: \"0afd3a3c-bf70-492b-ba79-489a2d0034ff\") "
Sep 17 16:01:42.686843 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:42.686819    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afd3a3c-bf70-492b-ba79-489a2d0034ff-kube-api-access-wf84t" (OuterVolumeSpecName: "kube-api-access-wf84t") pod "0afd3a3c-bf70-492b-ba79-489a2d0034ff" (UID: "0afd3a3c-bf70-492b-ba79-489a2d0034ff"). InnerVolumeSpecName "kube-api-access-wf84t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:01:42.786009 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:42.785985    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf84t\" (UniqueName: \"kubernetes.io/projected/0afd3a3c-bf70-492b-ba79-489a2d0034ff-kube-api-access-wf84t\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:01:43.432553 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:43.432524    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/curl-metrics-dcdff4d5"
Sep 17 16:01:43.432713 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:43.432531    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="542aa4d3e8050dc35a93a01aea139ce70debcfc5717dc340fb45c330dc33f6ba"
Sep 17 16:01:43.442154 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:01:43.442127    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"curl-metrics-dcdff4d5\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" podUID="0afd3a3c-bf70-492b-ba79-489a2d0034ff" pod="kserve/curl-metrics-dcdff4d5"
Sep 17 16:01:43.557313 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:43.557283    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0afd3a3c-bf70-492b-ba79-489a2d0034ff" path="/var/lib/kubelet/pods/0afd3a3c-bf70-492b-ba79-489a2d0034ff/volumes"
Sep 17 16:01:48.011333 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.011297    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"]
Sep 17 16:01:48.011725 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.011636    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0afd3a3c-bf70-492b-ba79-489a2d0034ff" containerName="curl-metrics-dcdff4d5"
Sep 17 16:01:48.011725 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.011648    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0afd3a3c-bf70-492b-ba79-489a2d0034ff" containerName="curl-metrics-dcdff4d5"
Sep 17 16:01:48.011725 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.011697    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0afd3a3c-bf70-492b-ba79-489a2d0034ff" containerName="curl-metrics-dcdff4d5"
Sep 17 16:01:48.015424 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.015407    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.017306 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.017270    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 17 16:01:48.017608 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.017593    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:01:48.018154 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.018135    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 17 16:01:48.018316 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.018298    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:01:48.018316 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.018301    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:01:48.018613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.018593    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:01:48.018678 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.018650    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:01:48.030351 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.030328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"]
Sep 17 16:01:48.134642 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.134608    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-cabundle-cert\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.134642 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.134644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kserve-provision-location\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.134835 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.134711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-proxy-tls\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.134835 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.134742    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.134835 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.134765    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5jnd\" (UniqueName: \"kubernetes.io/projected/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-api-access-b5jnd\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.235384 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.235337    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-proxy-tls\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.235384 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.235384    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.235646 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.235409    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5jnd\" (UniqueName: \"kubernetes.io/projected/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-api-access-b5jnd\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.235646 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.235493    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-cabundle-cert\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.235646 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.235550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kserve-provision-location\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.236001 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.235979    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kserve-provision-location\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.236227 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.236209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-cabundle-cert\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.236265 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.236245    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.237848 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.237830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-proxy-tls\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.243881 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.243860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5jnd\" (UniqueName: \"kubernetes.io/projected/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-api-access-b5jnd\") pod \"isvc-sklearn-predictor-58f64f5dbf-hqww4\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.325992 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.325969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:48.605160 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:48.605136    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"]
Sep 17 16:01:48.607075 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:01:48.607049    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5a81f7_979b_4890_ba4c_5cc59c904ba0.slice/crio-17479f563916e85d77538b7ccf76f48e046f89cd3294d451f1e04a3a712bf4e8 WatchSource:0}: Error finding container 17479f563916e85d77538b7ccf76f48e046f89cd3294d451f1e04a3a712bf4e8: Status 404 returned error can't find the container with id 17479f563916e85d77538b7ccf76f48e046f89cd3294d451f1e04a3a712bf4e8
Sep 17 16:01:49.460581 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:49.460547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" event={"ID":"3f5a81f7-979b-4890-ba4c-5cc59c904ba0","Type":"ContainerStarted","Data":"0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745"}
Sep 17 16:01:49.460952 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:49.460602    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" event={"ID":"3f5a81f7-979b-4890-ba4c-5cc59c904ba0","Type":"ContainerStarted","Data":"17479f563916e85d77538b7ccf76f48e046f89cd3294d451f1e04a3a712bf4e8"}
Sep 17 16:01:53.479682 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:53.479646    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerID="0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745" exitCode=0
Sep 17 16:01:53.480128 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:53.479723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" event={"ID":"3f5a81f7-979b-4890-ba4c-5cc59c904ba0","Type":"ContainerDied","Data":"0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745"}
Sep 17 16:01:54.486112 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:54.486081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" event={"ID":"3f5a81f7-979b-4890-ba4c-5cc59c904ba0","Type":"ContainerStarted","Data":"2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3"}
Sep 17 16:01:54.486504 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:54.486142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" event={"ID":"3f5a81f7-979b-4890-ba4c-5cc59c904ba0","Type":"ContainerStarted","Data":"645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b"}
Sep 17 16:01:54.486504 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:54.486343    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:54.509188 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:54.509147    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podStartSLOduration=3.515642979 podStartE2EDuration="7.509133214s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.993490235s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:01:54.50821089 +0000 UTC m=+1405.431785482" watchObservedRunningTime="2026-09-17 16:01:54.509133214 +0000 UTC m=+1405.432707781"
Sep 17 16:01:55.490695 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:55.490660    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:01:55.491872 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:55.491844    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:01:56.495481 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:01:56.495427    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:02:01.500283 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:02:01.500251    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:02:01.500882 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:02:01.500856    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:02:11.501023 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:02:11.500975    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:02:21.501605 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:02:21.501567    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:02:29.672439 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:02:29.672408    2581 scope.go:117] "RemoveContainer" containerID="f9b96e032cb1734329691aa81b56a68b1aecbf98ce325a6c2bd3eb818aad3183"
Sep 17 16:02:31.501191 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:02:31.501147    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:02:41.501691 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:02:41.501648    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:02:51.501087 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:02:51.501051    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:03:01.500934 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:01.500892    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:03:11.502310 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:11.502275    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:03:18.098660 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:18.098626    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"]
Sep 17 16:03:18.099139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:18.098938    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" containerID="cri-o://645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b" gracePeriod=30
Sep 17 16:03:18.099139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:18.098957    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kube-rbac-proxy" containerID="cri-o://2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3" gracePeriod=30
Sep 17 16:03:18.853142 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:18.853112    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerID="2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3" exitCode=2
Sep 17 16:03:18.853331 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:18.853191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" event={"ID":"3f5a81f7-979b-4890-ba4c-5cc59c904ba0","Type":"ContainerDied","Data":"2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3"}
Sep 17 16:03:21.496321 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:21.496270    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Sep 17 16:03:21.501693 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:21.501663    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Sep 17 16:03:23.038180 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.038156    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:03:23.142555 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.142520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-cabundle-cert\") pod \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") "
Sep 17 16:03:23.142768 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.142567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-rbac-proxy-sar-config\") pod \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") "
Sep 17 16:03:23.142768 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.142587    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kserve-provision-location\") pod \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") "
Sep 17 16:03:23.142768 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.142642    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-proxy-tls\") pod \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") "
Sep 17 16:03:23.142768 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.142660    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5jnd\" (UniqueName: \"kubernetes.io/projected/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-api-access-b5jnd\") pod \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\" (UID: \"3f5a81f7-979b-4890-ba4c-5cc59c904ba0\") "
Sep 17 16:03:23.143037 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.143007    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3f5a81f7-979b-4890-ba4c-5cc59c904ba0" (UID: "3f5a81f7-979b-4890-ba4c-5cc59c904ba0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:03:23.143096 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.143030    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3f5a81f7-979b-4890-ba4c-5cc59c904ba0" (UID: "3f5a81f7-979b-4890-ba4c-5cc59c904ba0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:23.143181 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.143161    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3f5a81f7-979b-4890-ba4c-5cc59c904ba0" (UID: "3f5a81f7-979b-4890-ba4c-5cc59c904ba0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:03:23.144804 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.144780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3f5a81f7-979b-4890-ba4c-5cc59c904ba0" (UID: "3f5a81f7-979b-4890-ba4c-5cc59c904ba0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:03:23.144922 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.144907    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-api-access-b5jnd" (OuterVolumeSpecName: "kube-api-access-b5jnd") pod "3f5a81f7-979b-4890-ba4c-5cc59c904ba0" (UID: "3f5a81f7-979b-4890-ba4c-5cc59c904ba0"). InnerVolumeSpecName "kube-api-access-b5jnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:03:23.244290 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.244204    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:03:23.244290 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.244229    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:03:23.244290 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.244238    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:03:23.244290 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.244247    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:03:23.244290 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.244255    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5jnd\" (UniqueName: \"kubernetes.io/projected/3f5a81f7-979b-4890-ba4c-5cc59c904ba0-kube-api-access-b5jnd\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:03:23.343488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.343441    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"]
Sep 17 16:03:23.343788 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.343775    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kube-rbac-proxy"
Sep 17 16:03:23.343848 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.343790    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kube-rbac-proxy"
Sep 17 16:03:23.343848 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.343807    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container"
Sep 17 16:03:23.343848 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.343812    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container"
Sep 17 16:03:23.343848 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.343824    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="storage-initializer"
Sep 17 16:03:23.343848 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.343831    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="storage-initializer"
Sep 17 16:03:23.344005 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.343876    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kube-rbac-proxy"
Sep 17 16:03:23.344005 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.343886    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerName="kserve-container"
Sep 17 16:03:23.347283 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.347251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.349649 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.349624    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 17 16:03:23.349991 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.349972    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 16:03:23.384715 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.384683    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"]
Sep 17 16:03:23.445946 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.445912    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.445946 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.445944    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.446211 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.445972    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.446211 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.446058    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.446211 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.446122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kdnd\" (UniqueName: \"kubernetes.io/projected/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-api-access-7kdnd\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.546814 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.546784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.546982 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.546819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.546982 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.546870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.547092 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.547026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.547092 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.547088    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kdnd\" (UniqueName: \"kubernetes.io/projected/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-api-access-7kdnd\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.547290 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.547268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.547630 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.547614    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.547683 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.547638    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.549326 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.549301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.560344 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.560324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kdnd\" (UniqueName: \"kubernetes.io/projected/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-api-access-7kdnd\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.657221 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.657185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:03:23.876138 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.876049    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" containerID="645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b" exitCode=0
Sep 17 16:03:23.876138 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.876108    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"
Sep 17 16:03:23.876442 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.876136    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" event={"ID":"3f5a81f7-979b-4890-ba4c-5cc59c904ba0","Type":"ContainerDied","Data":"645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b"}
Sep 17 16:03:23.876442 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.876207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4" event={"ID":"3f5a81f7-979b-4890-ba4c-5cc59c904ba0","Type":"ContainerDied","Data":"17479f563916e85d77538b7ccf76f48e046f89cd3294d451f1e04a3a712bf4e8"}
Sep 17 16:03:23.876442 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.876244    2581 scope.go:117] "RemoveContainer" containerID="2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3"
Sep 17 16:03:23.884197 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.884173    2581 scope.go:117] "RemoveContainer" containerID="645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b"
Sep 17 16:03:23.891805 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.891782    2581 scope.go:117] "RemoveContainer" containerID="0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745"
Sep 17 16:03:23.899338 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.899316    2581 scope.go:117] "RemoveContainer" containerID="2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3"
Sep 17 16:03:23.899652 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:03:23.899633    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3\": container with ID starting with 2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3 not found: ID does not exist" containerID="2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3"
Sep 17 16:03:23.899735 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.899661    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3"} err="failed to get container status \"2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3\": rpc error: code = NotFound desc = could not find container \"2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3\": container with ID starting with 2b288417b2640550e57da448ca9c0831705b0d3ed2c93ac3cd2122439a883dd3 not found: ID does not exist"
Sep 17 16:03:23.899735 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.899680    2581 scope.go:117] "RemoveContainer" containerID="645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b"
Sep 17 16:03:23.899954 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:03:23.899926    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b\": container with ID starting with 645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b not found: ID does not exist" containerID="645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b"
Sep 17 16:03:23.900025 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.899964    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b"} err="failed to get container status \"645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b\": rpc error: code = NotFound desc = could not find container \"645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b\": container with ID starting with 645ef36633b2fc405d88f5d348a7ff11647dd95c376cad8020a3a515c5d2704b not found: ID does not exist"
Sep 17 16:03:23.900025 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.899984    2581 scope.go:117] "RemoveContainer" containerID="0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745"
Sep 17 16:03:23.900237 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.900219    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"]
Sep 17 16:03:23.900288 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:03:23.900265    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745\": container with ID starting with 0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745 not found: ID does not exist" containerID="0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745"
Sep 17 16:03:23.900323 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.900282    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745"} err="failed to get container status \"0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745\": rpc error: code = NotFound desc = could not find container \"0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745\": container with ID starting with 0fb036692d0d5dbee301fc20e3cd6918164b16307345e7ed0811f7c2c5117745 not found: ID does not exist"
Sep 17 16:03:23.902751 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.902726    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-58f64f5dbf-hqww4"]
Sep 17 16:03:23.938400 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:23.938363    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"]
Sep 17 16:03:23.941706 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:03:23.941674    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c0ab68_d563_461a_a2e6_607bc4e5c3e3.slice/crio-ef27748fda0fc48a5c737050f17d974929fdfd3e9c8f29729165d15376fd9f9b WatchSource:0}: Error finding container ef27748fda0fc48a5c737050f17d974929fdfd3e9c8f29729165d15376fd9f9b: Status 404 returned error can't find the container with id ef27748fda0fc48a5c737050f17d974929fdfd3e9c8f29729165d15376fd9f9b
Sep 17 16:03:24.882821 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:24.882787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" event={"ID":"85c0ab68-d563-461a-a2e6-607bc4e5c3e3","Type":"ContainerStarted","Data":"1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96"}
Sep 17 16:03:24.883230 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:24.882844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" event={"ID":"85c0ab68-d563-461a-a2e6-607bc4e5c3e3","Type":"ContainerStarted","Data":"ef27748fda0fc48a5c737050f17d974929fdfd3e9c8f29729165d15376fd9f9b"}
Sep 17 16:03:25.557177 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:25.557143    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f5a81f7-979b-4890-ba4c-5cc59c904ba0" path="/var/lib/kubelet/pods/3f5a81f7-979b-4890-ba4c-5cc59c904ba0/volumes"
Sep 17 16:03:28.906141 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:28.906103    2581 generic.go:356] "Generic (PLEG): container finished" podID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerID="1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96" exitCode=0
Sep 17 16:03:28.906594 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:28.906164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" event={"ID":"85c0ab68-d563-461a-a2e6-607bc4e5c3e3","Type":"ContainerDied","Data":"1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96"}
Sep 17 16:03:29.690954 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:29.690819    2581 scope.go:117] "RemoveContainer" containerID="514f30b23ca7e7220ba19d51b69a5fd4d18aab0aabeae1c9596d74c3767f2b01"
Sep 17 16:03:29.714065 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:03:29.713926    2581 scope.go:117] "RemoveContainer" containerID="a747151c6aed3fd994a28cb2c59733a721017d8ba1919eb0cb4c0ee3168a7ca1"
Sep 17 16:05:41.570438 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:05:41.570400    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" event={"ID":"85c0ab68-d563-461a-a2e6-607bc4e5c3e3","Type":"ContainerStarted","Data":"f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea"}
Sep 17 16:05:41.570868 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:05:41.570455    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" event={"ID":"85c0ab68-d563-461a-a2e6-607bc4e5c3e3","Type":"ContainerStarted","Data":"9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2"}
Sep 17 16:05:41.570868 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:05:41.570506    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:05:41.570868 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:05:41.570542    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:05:41.592981 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:05:41.592924    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" podStartSLOduration=2.455036183 podStartE2EDuration="2m18.592910743s" totalImagesPullingTime="2m12.20987155s" totalInitContainerRuntime="3.92800301s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:03:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:05:41.591856143 +0000 UTC m=+1632.515430709" watchObservedRunningTime="2026-09-17 16:05:41.592910743 +0000 UTC m=+1632.516485309"
Sep 17 16:05:47.580481 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:05:47.580442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:06:17.614748 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:17.614692    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 16:06:27.583672 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:27.583641    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:06:33.433950 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:33.433910    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"]
Sep 17 16:06:33.434423 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:33.434194    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kserve-container" containerID="cri-o://9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2" gracePeriod=30
Sep 17 16:06:33.434423 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:33.434236    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kube-rbac-proxy" containerID="cri-o://f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea" gracePeriod=30
Sep 17 16:06:33.794237 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:33.794199    2581 generic.go:356] "Generic (PLEG): container finished" podID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerID="f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea" exitCode=2
Sep 17 16:06:33.794237 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:33.794237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" event={"ID":"85c0ab68-d563-461a-a2e6-607bc4e5c3e3","Type":"ContainerDied","Data":"f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea"}
Sep 17 16:06:37.576036 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:37.575995    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Sep 17 16:06:37.581086 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:37.581055    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/sklearn-v2-mlserver/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 17 16:06:38.684978 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.684946    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"]
Sep 17 16:06:38.688523 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.688506    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.690407 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.690384    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 17 16:06:38.690407 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.690398    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:06:38.699764 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.699740    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"]
Sep 17 16:06:38.803348 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.803309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjx87\" (UniqueName: \"kubernetes.io/projected/d7522e72-f36f-438d-adac-b52867709c9c-kube-api-access-fjx87\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.803544 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.803396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7522e72-f36f-438d-adac-b52867709c9c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.803544 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.803426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.803544 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.803455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7522e72-f36f-438d-adac-b52867709c9c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.803544 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.803532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.904645 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.904599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjx87\" (UniqueName: \"kubernetes.io/projected/d7522e72-f36f-438d-adac-b52867709c9c-kube-api-access-fjx87\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.904860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.904689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7522e72-f36f-438d-adac-b52867709c9c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.904860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.904722    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.904860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.904760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7522e72-f36f-438d-adac-b52867709c9c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.904860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.904783    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.905139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.905112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7522e72-f36f-438d-adac-b52867709c9c-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.905487 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.905439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.905611 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.905458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.907286 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.907270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7522e72-f36f-438d-adac-b52867709c9c-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.913877 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.913856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjx87\" (UniqueName: \"kubernetes.io/projected/d7522e72-f36f-438d-adac-b52867709c9c-kube-api-access-fjx87\") pod \"isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:38.998345 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:38.998255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:39.273830 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:39.273801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"]
Sep 17 16:06:39.276446 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:06:39.276414    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7522e72_f36f_438d_adac_b52867709c9c.slice/crio-d3f72f2907d0b6ed04d2a25d1f79c6b9962ee4b2734abcec168a70169b1fb854 WatchSource:0}: Error finding container d3f72f2907d0b6ed04d2a25d1f79c6b9962ee4b2734abcec168a70169b1fb854: Status 404 returned error can't find the container with id d3f72f2907d0b6ed04d2a25d1f79c6b9962ee4b2734abcec168a70169b1fb854
Sep 17 16:06:39.825633 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:39.825596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" event={"ID":"d7522e72-f36f-438d-adac-b52867709c9c","Type":"ContainerStarted","Data":"4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224"}
Sep 17 16:06:39.826032 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:39.825686    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" event={"ID":"d7522e72-f36f-438d-adac-b52867709c9c","Type":"ContainerStarted","Data":"d3f72f2907d0b6ed04d2a25d1f79c6b9962ee4b2734abcec168a70169b1fb854"}
Sep 17 16:06:41.278272 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.278247    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:06:41.323952 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.323918    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-proxy-tls\") pod \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") "
Sep 17 16:06:41.324146 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.323963    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-cabundle-cert\") pod \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") "
Sep 17 16:06:41.324146 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.323990    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kserve-provision-location\") pod \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") "
Sep 17 16:06:41.324146 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.324007    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kdnd\" (UniqueName: \"kubernetes.io/projected/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-api-access-7kdnd\") pod \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") "
Sep 17 16:06:41.324146 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.324030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-rbac-proxy-sar-config\") pod \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\" (UID: \"85c0ab68-d563-461a-a2e6-607bc4e5c3e3\") "
Sep 17 16:06:41.324401 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.324370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "85c0ab68-d563-461a-a2e6-607bc4e5c3e3" (UID: "85c0ab68-d563-461a-a2e6-607bc4e5c3e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:06:41.324458 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.324412    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "85c0ab68-d563-461a-a2e6-607bc4e5c3e3" (UID: "85c0ab68-d563-461a-a2e6-607bc4e5c3e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:06:41.324543 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.324520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "85c0ab68-d563-461a-a2e6-607bc4e5c3e3" (UID: "85c0ab68-d563-461a-a2e6-607bc4e5c3e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:06:41.326086 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.326066    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-api-access-7kdnd" (OuterVolumeSpecName: "kube-api-access-7kdnd") pod "85c0ab68-d563-461a-a2e6-607bc4e5c3e3" (UID: "85c0ab68-d563-461a-a2e6-607bc4e5c3e3"). InnerVolumeSpecName "kube-api-access-7kdnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:06:41.326139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.326118    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "85c0ab68-d563-461a-a2e6-607bc4e5c3e3" (UID: "85c0ab68-d563-461a-a2e6-607bc4e5c3e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:06:41.425189 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.425101    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:06:41.425189 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.425128    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:06:41.425189 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.425138    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:06:41.425189 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.425147    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kdnd\" (UniqueName: \"kubernetes.io/projected/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-api-access-7kdnd\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:06:41.425189 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.425157    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/85c0ab68-d563-461a-a2e6-607bc4e5c3e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:06:41.836052 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.836020    2581 generic.go:356] "Generic (PLEG): container finished" podID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerID="9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2" exitCode=0
Sep 17 16:06:41.836239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.836080    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"
Sep 17 16:06:41.836239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.836081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" event={"ID":"85c0ab68-d563-461a-a2e6-607bc4e5c3e3","Type":"ContainerDied","Data":"9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2"}
Sep 17 16:06:41.836239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.836207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k" event={"ID":"85c0ab68-d563-461a-a2e6-607bc4e5c3e3","Type":"ContainerDied","Data":"ef27748fda0fc48a5c737050f17d974929fdfd3e9c8f29729165d15376fd9f9b"}
Sep 17 16:06:41.836239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.836238    2581 scope.go:117] "RemoveContainer" containerID="f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea"
Sep 17 16:06:41.844600 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.844582    2581 scope.go:117] "RemoveContainer" containerID="9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2"
Sep 17 16:06:41.852087 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.852056    2581 scope.go:117] "RemoveContainer" containerID="1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96"
Sep 17 16:06:41.853858 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.853836    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"]
Sep 17 16:06:41.859993 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.859972    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-hfg6k"]
Sep 17 16:06:41.860146 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.860133    2581 scope.go:117] "RemoveContainer" containerID="f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea"
Sep 17 16:06:41.860432 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:06:41.860416    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea\": container with ID starting with f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea not found: ID does not exist" containerID="f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea"
Sep 17 16:06:41.860488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.860441    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea"} err="failed to get container status \"f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea\": rpc error: code = NotFound desc = could not find container \"f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea\": container with ID starting with f47bfbd7f21df502a0733f59e094e69b38de6cc93618d903fa392fdfcb21f1ea not found: ID does not exist"
Sep 17 16:06:41.860488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.860478    2581 scope.go:117] "RemoveContainer" containerID="9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2"
Sep 17 16:06:41.860757 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:06:41.860740    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2\": container with ID starting with 9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2 not found: ID does not exist" containerID="9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2"
Sep 17 16:06:41.860793 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.860763    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2"} err="failed to get container status \"9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2\": rpc error: code = NotFound desc = could not find container \"9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2\": container with ID starting with 9df01218e13f2e61c004d15ed4c155a89b1e55eaa699e6b2fb6fa461a8437ff2 not found: ID does not exist"
Sep 17 16:06:41.860793 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.860778    2581 scope.go:117] "RemoveContainer" containerID="1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96"
Sep 17 16:06:41.861046 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:06:41.861030    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96\": container with ID starting with 1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96 not found: ID does not exist" containerID="1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96"
Sep 17 16:06:41.861082 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:41.861060    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96"} err="failed to get container status \"1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96\": rpc error: code = NotFound desc = could not find container \"1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96\": container with ID starting with 1bd5af8e280df326f122b4602b6f9a31b2465bcba9444d8bdb0972e2fd6c5a96 not found: ID does not exist"
Sep 17 16:06:43.558594 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:43.558552    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" path="/var/lib/kubelet/pods/85c0ab68-d563-461a-a2e6-607bc4e5c3e3/volumes"
Sep 17 16:06:45.860129 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:45.860097    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7522e72-f36f-438d-adac-b52867709c9c" containerID="4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224" exitCode=0
Sep 17 16:06:45.860533 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:45.860174    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" event={"ID":"d7522e72-f36f-438d-adac-b52867709c9c","Type":"ContainerDied","Data":"4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224"}
Sep 17 16:06:46.873643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:46.873598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" event={"ID":"d7522e72-f36f-438d-adac-b52867709c9c","Type":"ContainerStarted","Data":"35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e"}
Sep 17 16:06:46.874010 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:46.873716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" event={"ID":"d7522e72-f36f-438d-adac-b52867709c9c","Type":"ContainerStarted","Data":"81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd"}
Sep 17 16:06:46.874010 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:46.873925    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:46.896417 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:46.896333    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" podStartSLOduration=3.350055356 podStartE2EDuration="8.896313432s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.546258076s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:06:46.895170342 +0000 UTC m=+1697.818744910" watchObservedRunningTime="2026-09-17 16:06:46.896313432 +0000 UTC m=+1697.819887999"
Sep 17 16:06:47.878590 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:47.878545    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:47.879820 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:47.879792    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:06:48.884694 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:48.884660    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:06:53.889115 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:53.889044    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:06:53.889595 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:53.889569    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 17 16:06:56.159749 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.159716    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"]
Sep 17 16:06:56.160140 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.160069    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kube-rbac-proxy"
Sep 17 16:06:56.160140 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.160081    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kube-rbac-proxy"
Sep 17 16:06:56.160140 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.160101    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="storage-initializer"
Sep 17 16:06:56.160140 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.160107    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="storage-initializer"
Sep 17 16:06:56.160140 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.160116    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kserve-container"
Sep 17 16:06:56.160140 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.160121    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kserve-container"
Sep 17 16:06:56.160334 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.160181    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kube-rbac-proxy"
Sep 17 16:06:56.160334 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.160191    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c0ab68-d563-461a-a2e6-607bc4e5c3e3" containerName="kserve-container"
Sep 17 16:06:56.163586 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.163562    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.165649 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.165628    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 16:06:56.165774 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.165674    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 16:06:56.173181 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.173156    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"]
Sep 17 16:06:56.261820 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.261780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.261820 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.261821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.262052 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.261927    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0efe4df5-42b3-4baa-acbc-7945817ccbb3-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.262052 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.261955    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6nrd\" (UniqueName: \"kubernetes.io/projected/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-api-access-j6nrd\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.262052 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.261974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.362557 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.362516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.362557 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.362559    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.362827 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.362611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0efe4df5-42b3-4baa-acbc-7945817ccbb3-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.362827 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.362648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6nrd\" (UniqueName: \"kubernetes.io/projected/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-api-access-j6nrd\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.362827 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.362674    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.363085 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.363065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kserve-provision-location\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.363289 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.363264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-cabundle-cert\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.363368 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.363289    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.365363 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.365329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0efe4df5-42b3-4baa-acbc-7945817ccbb3-proxy-tls\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.371740 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.371719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6nrd\" (UniqueName: \"kubernetes.io/projected/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-api-access-j6nrd\") pod \"isvc-canary-create-predictor-5674b8c456-rd6t5\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.477157 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.477072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:06:56.764719 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.764635    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"]
Sep 17 16:06:56.767498 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:06:56.767449    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efe4df5_42b3_4baa_acbc_7945817ccbb3.slice/crio-baa8909ba1a002f7061d11fd42f157d0584235c795717ca6bd61ffb434b5bd2c WatchSource:0}: Error finding container baa8909ba1a002f7061d11fd42f157d0584235c795717ca6bd61ffb434b5bd2c: Status 404 returned error can't find the container with id baa8909ba1a002f7061d11fd42f157d0584235c795717ca6bd61ffb434b5bd2c
Sep 17 16:06:56.924526 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.924488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" event={"ID":"0efe4df5-42b3-4baa-acbc-7945817ccbb3","Type":"ContainerStarted","Data":"2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5"}
Sep 17 16:06:56.924748 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:06:56.924556    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" event={"ID":"0efe4df5-42b3-4baa-acbc-7945817ccbb3","Type":"ContainerStarted","Data":"baa8909ba1a002f7061d11fd42f157d0584235c795717ca6bd61ffb434b5bd2c"}
Sep 17 16:07:00.948577 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:00.948545    2581 generic.go:356] "Generic (PLEG): container finished" podID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerID="2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5" exitCode=0
Sep 17 16:07:00.948998 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:00.948631    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" event={"ID":"0efe4df5-42b3-4baa-acbc-7945817ccbb3","Type":"ContainerDied","Data":"2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5"}
Sep 17 16:07:01.956362 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:01.956326    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" event={"ID":"0efe4df5-42b3-4baa-acbc-7945817ccbb3","Type":"ContainerStarted","Data":"6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917"}
Sep 17 16:07:01.956790 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:01.956394    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" event={"ID":"0efe4df5-42b3-4baa-acbc-7945817ccbb3","Type":"ContainerStarted","Data":"316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b"}
Sep 17 16:07:01.956790 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:01.956604    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:07:01.977436 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:01.977384    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podStartSLOduration=1.940688641 podStartE2EDuration="5.977364168s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.036675527s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:01.975663715 +0000 UTC m=+1712.899238282" watchObservedRunningTime="2026-09-17 16:07:01.977364168 +0000 UTC m=+1712.900938792"
Sep 17 16:07:02.961328 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:02.961293    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:07:02.962756 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:02.962725    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:07:03.890022 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:03.889985    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:07:03.966599 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:03.966561    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:07:08.971272 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:08.971197    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:07:08.971799 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:08.971771    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:07:10.607245 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:10.607211    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"]
Sep 17 16:07:10.607654 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:10.607439    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kserve-container" containerID="cri-o://81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd" gracePeriod=30
Sep 17 16:07:10.607654 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:10.607503    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kube-rbac-proxy" containerID="cri-o://35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e" gracePeriod=30
Sep 17 16:07:11.001358 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.001278    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7522e72-f36f-438d-adac-b52867709c9c" containerID="35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e" exitCode=2
Sep 17 16:07:11.001358 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.001347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" event={"ID":"d7522e72-f36f-438d-adac-b52867709c9c","Type":"ContainerDied","Data":"35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e"}
Sep 17 16:07:11.555779 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.555755    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:07:11.602388 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.602349    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-kube-rbac-proxy-sar-config\") pod \"d7522e72-f36f-438d-adac-b52867709c9c\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") "
Sep 17 16:07:11.602577 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.602394    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-cabundle-cert\") pod \"d7522e72-f36f-438d-adac-b52867709c9c\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") "
Sep 17 16:07:11.602577 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.602478    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7522e72-f36f-438d-adac-b52867709c9c-proxy-tls\") pod \"d7522e72-f36f-438d-adac-b52867709c9c\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") "
Sep 17 16:07:11.602577 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.602513    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjx87\" (UniqueName: \"kubernetes.io/projected/d7522e72-f36f-438d-adac-b52867709c9c-kube-api-access-fjx87\") pod \"d7522e72-f36f-438d-adac-b52867709c9c\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") "
Sep 17 16:07:11.602745 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.602577    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7522e72-f36f-438d-adac-b52867709c9c-kserve-provision-location\") pod \"d7522e72-f36f-438d-adac-b52867709c9c\" (UID: \"d7522e72-f36f-438d-adac-b52867709c9c\") "
Sep 17 16:07:11.602745 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.602727    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d7522e72-f36f-438d-adac-b52867709c9c" (UID: "d7522e72-f36f-438d-adac-b52867709c9c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:07:11.602828 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.602787    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d7522e72-f36f-438d-adac-b52867709c9c" (UID: "d7522e72-f36f-438d-adac-b52867709c9c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:07:11.602875 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.602835    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:07:11.604630 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.604598    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7522e72-f36f-438d-adac-b52867709c9c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d7522e72-f36f-438d-adac-b52867709c9c" (UID: "d7522e72-f36f-438d-adac-b52867709c9c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:07:11.604737 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.604657    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7522e72-f36f-438d-adac-b52867709c9c-kube-api-access-fjx87" (OuterVolumeSpecName: "kube-api-access-fjx87") pod "d7522e72-f36f-438d-adac-b52867709c9c" (UID: "d7522e72-f36f-438d-adac-b52867709c9c"). InnerVolumeSpecName "kube-api-access-fjx87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:07:11.629268 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.629233    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7522e72-f36f-438d-adac-b52867709c9c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d7522e72-f36f-438d-adac-b52867709c9c" (UID: "d7522e72-f36f-438d-adac-b52867709c9c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:07:11.703665 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.703588    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d7522e72-f36f-438d-adac-b52867709c9c-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:07:11.703665 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.703621    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d7522e72-f36f-438d-adac-b52867709c9c-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:07:11.703665 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.703630    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d7522e72-f36f-438d-adac-b52867709c9c-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:07:11.703665 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:11.703639    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjx87\" (UniqueName: \"kubernetes.io/projected/d7522e72-f36f-438d-adac-b52867709c9c-kube-api-access-fjx87\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:07:12.008061 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.007975    2581 generic.go:356] "Generic (PLEG): container finished" podID="d7522e72-f36f-438d-adac-b52867709c9c" containerID="81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd" exitCode=0
Sep 17 16:07:12.008061 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.008033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" event={"ID":"d7522e72-f36f-438d-adac-b52867709c9c","Type":"ContainerDied","Data":"81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd"}
Sep 17 16:07:12.008295 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.008079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5" event={"ID":"d7522e72-f36f-438d-adac-b52867709c9c","Type":"ContainerDied","Data":"d3f72f2907d0b6ed04d2a25d1f79c6b9962ee4b2734abcec168a70169b1fb854"}
Sep 17 16:07:12.008295 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.008035    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"
Sep 17 16:07:12.008295 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.008112    2581 scope.go:117] "RemoveContainer" containerID="35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e"
Sep 17 16:07:12.017978 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.017963    2581 scope.go:117] "RemoveContainer" containerID="81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd"
Sep 17 16:07:12.025408 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.025390    2581 scope.go:117] "RemoveContainer" containerID="4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224"
Sep 17 16:07:12.032941 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.032924    2581 scope.go:117] "RemoveContainer" containerID="35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e"
Sep 17 16:07:12.033177 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:07:12.033160    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e\": container with ID starting with 35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e not found: ID does not exist" containerID="35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e"
Sep 17 16:07:12.033241 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.033192    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"]
Sep 17 16:07:12.033241 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.033188    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e"} err="failed to get container status \"35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e\": rpc error: code = NotFound desc = could not find container \"35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e\": container with ID starting with 35ef9b8f37d71397afebd4ab0ca9aff113316225bf5ca639933d6218bfe4eb4e not found: ID does not exist"
Sep 17 16:07:12.033241 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.033233    2581 scope.go:117] "RemoveContainer" containerID="81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd"
Sep 17 16:07:12.033523 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:07:12.033507    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd\": container with ID starting with 81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd not found: ID does not exist" containerID="81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd"
Sep 17 16:07:12.033572 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.033530    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd"} err="failed to get container status \"81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd\": rpc error: code = NotFound desc = could not find container \"81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd\": container with ID starting with 81e8d119ee58923c9c9450d9994abed03347709065f278300d83419f1d7e25bd not found: ID does not exist"
Sep 17 16:07:12.033572 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.033546    2581 scope.go:117] "RemoveContainer" containerID="4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224"
Sep 17 16:07:12.033797 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:07:12.033779    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224\": container with ID starting with 4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224 not found: ID does not exist" containerID="4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224"
Sep 17 16:07:12.033853 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.033801    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224"} err="failed to get container status \"4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224\": rpc error: code = NotFound desc = could not find container \"4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224\": container with ID starting with 4cdf3027c778ea44bbcc35c01eea1cb7fc7b5bf4a99a94a145c31554708e1224 not found: ID does not exist"
Sep 17 16:07:12.041436 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:12.041417    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-56d59b6d5c-tw7p5"]
Sep 17 16:07:13.557508 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:13.557458    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7522e72-f36f-438d-adac-b52867709c9c" path="/var/lib/kubelet/pods/d7522e72-f36f-438d-adac-b52867709c9c/volumes"
Sep 17 16:07:15.848558 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.848524    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"]
Sep 17 16:07:15.848926 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.848862    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kube-rbac-proxy"
Sep 17 16:07:15.848926 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.848873    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kube-rbac-proxy"
Sep 17 16:07:15.848926 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.848887    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kserve-container"
Sep 17 16:07:15.848926 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.848893    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kserve-container"
Sep 17 16:07:15.848926 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.848911    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="storage-initializer"
Sep 17 16:07:15.848926 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.848918    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="storage-initializer"
Sep 17 16:07:15.849133 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.848966    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kserve-container"
Sep 17 16:07:15.849133 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.848977    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7522e72-f36f-438d-adac-b52867709c9c" containerName="kube-rbac-proxy"
Sep 17 16:07:15.853536 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.853518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:15.857912 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.857889    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 17 16:07:15.858039 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.857913    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:07:15.866819 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.866797    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"]
Sep 17 16:07:15.938919 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.938866    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:15.938919 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.938921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qxj\" (UniqueName: \"kubernetes.io/projected/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-api-access-88qxj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:15.939154 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.938949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77855d5b-37fc-4147-9510-6c831f10bbf5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:15.939154 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.938971    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:15.939154 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:15.939003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77855d5b-37fc-4147-9510-6c831f10bbf5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.040521 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.040449    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.040672 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.040549    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qxj\" (UniqueName: \"kubernetes.io/projected/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-api-access-88qxj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.040672 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.040606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77855d5b-37fc-4147-9510-6c831f10bbf5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.040672 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.040647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.040855 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.040712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77855d5b-37fc-4147-9510-6c831f10bbf5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.041231 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.041204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.041412 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.041388    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77855d5b-37fc-4147-9510-6c831f10bbf5-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.041478 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.041425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.043109 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.043090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77855d5b-37fc-4147-9510-6c831f10bbf5-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.055259 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.055230    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qxj\" (UniqueName: \"kubernetes.io/projected/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-api-access-88qxj\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.163239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.163148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:16.438538 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:16.438447    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"]
Sep 17 16:07:16.442171 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:07:16.442141    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77855d5b_37fc_4147_9510_6c831f10bbf5.slice/crio-fc1511461e4874d6e3a4c92434e0052cf161b5da0fd84b38cc9f6d63afd7d26b WatchSource:0}: Error finding container fc1511461e4874d6e3a4c92434e0052cf161b5da0fd84b38cc9f6d63afd7d26b: Status 404 returned error can't find the container with id fc1511461e4874d6e3a4c92434e0052cf161b5da0fd84b38cc9f6d63afd7d26b
Sep 17 16:07:17.038078 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:17.038043    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" event={"ID":"77855d5b-37fc-4147-9510-6c831f10bbf5","Type":"ContainerStarted","Data":"7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458"}
Sep 17 16:07:17.038530 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:17.038104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" event={"ID":"77855d5b-37fc-4147-9510-6c831f10bbf5","Type":"ContainerStarted","Data":"fc1511461e4874d6e3a4c92434e0052cf161b5da0fd84b38cc9f6d63afd7d26b"}
Sep 17 16:07:18.972231 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:18.972193    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:07:21.058099 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:21.058069    2581 generic.go:356] "Generic (PLEG): container finished" podID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerID="7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458" exitCode=0
Sep 17 16:07:21.058623 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:21.058128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" event={"ID":"77855d5b-37fc-4147-9510-6c831f10bbf5","Type":"ContainerDied","Data":"7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458"}
Sep 17 16:07:22.064963 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:22.064930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" event={"ID":"77855d5b-37fc-4147-9510-6c831f10bbf5","Type":"ContainerStarted","Data":"d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae"}
Sep 17 16:07:22.065354 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:22.064988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" event={"ID":"77855d5b-37fc-4147-9510-6c831f10bbf5","Type":"ContainerStarted","Data":"896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691"}
Sep 17 16:07:22.065354 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:22.065193    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:22.065354 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:22.065345    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:22.087302 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:22.087252    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" podStartSLOduration=2.961577855 podStartE2EDuration="7.087239165s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.12566131s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:07:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:07:22.085736465 +0000 UTC m=+1733.009311033" watchObservedRunningTime="2026-09-17 16:07:22.087239165 +0000 UTC m=+1733.010813731"
Sep 17 16:07:28.074780 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:28.074749    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:07:28.971830 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:28.971784    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:07:38.972508 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:38.972439    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:07:48.972081 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:48.972041    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:07:58.114850 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:58.114801    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 16:07:58.971979 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:07:58.971941    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:08:08.077723 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:08.077693    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:08:08.971772 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:08.971736    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:08:15.882175 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:15.882141    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"]
Sep 17 16:08:15.882746 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:15.882444    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kserve-container" containerID="cri-o://896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691" gracePeriod=30
Sep 17 16:08:15.882746 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:15.882504    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kube-rbac-proxy" containerID="cri-o://d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae" gracePeriod=30
Sep 17 16:08:16.302670 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:16.302642    2581 generic.go:356] "Generic (PLEG): container finished" podID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerID="d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae" exitCode=2
Sep 17 16:08:16.302841 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:16.302718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" event={"ID":"77855d5b-37fc-4147-9510-6c831f10bbf5","Type":"ContainerDied","Data":"d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae"}
Sep 17 16:08:18.070333 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:18.070282    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 16:08:18.972619 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:18.972589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:08:19.075937 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:19.075893    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-sklearn-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 16:08:21.155082 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.155047    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"]
Sep 17 16:08:21.158699 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.158680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.161420 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.161395    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:08:21.161956 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.161932    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 16:08:21.172302 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.172274    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"]
Sep 17 16:08:21.204820 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.204783    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.205012 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.204839    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbc2636-1222-4db9-a86b-df2f974c3f64-proxy-tls\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.205012 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.204920    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxn8n\" (UniqueName: \"kubernetes.io/projected/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-api-access-dxn8n\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.205012 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.204949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.205012 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.204970    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dbc2636-1222-4db9-a86b-df2f974c3f64-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.305798 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.305756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbc2636-1222-4db9-a86b-df2f974c3f64-proxy-tls\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.305987 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.305823    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxn8n\" (UniqueName: \"kubernetes.io/projected/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-api-access-dxn8n\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.305987 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.305851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.305987 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.305876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dbc2636-1222-4db9-a86b-df2f974c3f64-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.305987 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:08:21.305926    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 17 16:08:21.305987 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.305935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.306251 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:08:21.305995    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dbc2636-1222-4db9-a86b-df2f974c3f64-proxy-tls podName:2dbc2636-1222-4db9-a86b-df2f974c3f64 nodeName:}" failed. No retries permitted until 2026-09-17 16:08:21.805972195 +0000 UTC m=+1792.729546757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2dbc2636-1222-4db9-a86b-df2f974c3f64-proxy-tls") pod "isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" (UID: "2dbc2636-1222-4db9-a86b-df2f974c3f64") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 17 16:08:21.306416 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.306395    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dbc2636-1222-4db9-a86b-df2f974c3f64-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.306558 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.306540    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.306652 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.306636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.318204 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.318175    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxn8n\" (UniqueName: \"kubernetes.io/projected/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-api-access-dxn8n\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.810818 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.810784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbc2636-1222-4db9-a86b-df2f974c3f64-proxy-tls\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:21.813187 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:21.813155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbc2636-1222-4db9-a86b-df2f974c3f64-proxy-tls\") pod \"isvc-sklearn-v2-predictor-68f4f4d797-c6nvr\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:22.069010 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:22.068929    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:22.345346 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:22.345322    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"]
Sep 17 16:08:22.347713 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:08:22.347675    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dbc2636_1222_4db9_a86b_df2f974c3f64.slice/crio-9b01ebb2c97095b10d1bb1827e4c4e179bd99824def4df1f6af42c4f50e128f4 WatchSource:0}: Error finding container 9b01ebb2c97095b10d1bb1827e4c4e179bd99824def4df1f6af42c4f50e128f4: Status 404 returned error can't find the container with id 9b01ebb2c97095b10d1bb1827e4c4e179bd99824def4df1f6af42c4f50e128f4
Sep 17 16:08:23.070393 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.070337    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 17 16:08:23.336647 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.336616    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" event={"ID":"2dbc2636-1222-4db9-a86b-df2f974c3f64","Type":"ContainerStarted","Data":"ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296"}
Sep 17 16:08:23.336798 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.336699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" event={"ID":"2dbc2636-1222-4db9-a86b-df2f974c3f64","Type":"ContainerStarted","Data":"9b01ebb2c97095b10d1bb1827e4c4e179bd99824def4df1f6af42c4f50e128f4"}
Sep 17 16:08:23.523716 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.523692    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:08:23.626596 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.626503    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77855d5b-37fc-4147-9510-6c831f10bbf5-kserve-provision-location\") pod \"77855d5b-37fc-4147-9510-6c831f10bbf5\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") "
Sep 17 16:08:23.626596 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.626553    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88qxj\" (UniqueName: \"kubernetes.io/projected/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-api-access-88qxj\") pod \"77855d5b-37fc-4147-9510-6c831f10bbf5\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") "
Sep 17 16:08:23.626817 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.626598    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-rbac-proxy-sar-config\") pod \"77855d5b-37fc-4147-9510-6c831f10bbf5\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") "
Sep 17 16:08:23.626817 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.626666    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77855d5b-37fc-4147-9510-6c831f10bbf5-proxy-tls\") pod \"77855d5b-37fc-4147-9510-6c831f10bbf5\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") "
Sep 17 16:08:23.626817 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.626701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-cabundle-cert\") pod \"77855d5b-37fc-4147-9510-6c831f10bbf5\" (UID: \"77855d5b-37fc-4147-9510-6c831f10bbf5\") "
Sep 17 16:08:23.626981 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.626882    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77855d5b-37fc-4147-9510-6c831f10bbf5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "77855d5b-37fc-4147-9510-6c831f10bbf5" (UID: "77855d5b-37fc-4147-9510-6c831f10bbf5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:23.627038 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.626994    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77855d5b-37fc-4147-9510-6c831f10bbf5-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:23.627133 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.627110    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "77855d5b-37fc-4147-9510-6c831f10bbf5" (UID: "77855d5b-37fc-4147-9510-6c831f10bbf5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:23.627184 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.627128    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "77855d5b-37fc-4147-9510-6c831f10bbf5" (UID: "77855d5b-37fc-4147-9510-6c831f10bbf5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:23.628797 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.628775    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77855d5b-37fc-4147-9510-6c831f10bbf5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77855d5b-37fc-4147-9510-6c831f10bbf5" (UID: "77855d5b-37fc-4147-9510-6c831f10bbf5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:23.628890 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.628818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-api-access-88qxj" (OuterVolumeSpecName: "kube-api-access-88qxj") pod "77855d5b-37fc-4147-9510-6c831f10bbf5" (UID: "77855d5b-37fc-4147-9510-6c831f10bbf5"). InnerVolumeSpecName "kube-api-access-88qxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:23.728051 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.728020    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77855d5b-37fc-4147-9510-6c831f10bbf5-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:23.728051 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.728046    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:23.728051 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.728057    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qxj\" (UniqueName: \"kubernetes.io/projected/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-api-access-88qxj\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:23.728286 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:23.728065    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77855d5b-37fc-4147-9510-6c831f10bbf5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:24.343545 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.343507    2581 generic.go:356] "Generic (PLEG): container finished" podID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerID="896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691" exitCode=0
Sep 17 16:08:24.343755 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.343573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"
Sep 17 16:08:24.343755 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.343580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" event={"ID":"77855d5b-37fc-4147-9510-6c831f10bbf5","Type":"ContainerDied","Data":"896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691"}
Sep 17 16:08:24.343755 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.343636    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94" event={"ID":"77855d5b-37fc-4147-9510-6c831f10bbf5","Type":"ContainerDied","Data":"fc1511461e4874d6e3a4c92434e0052cf161b5da0fd84b38cc9f6d63afd7d26b"}
Sep 17 16:08:24.343755 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.343662    2581 scope.go:117] "RemoveContainer" containerID="d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae"
Sep 17 16:08:24.352841 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.352826    2581 scope.go:117] "RemoveContainer" containerID="896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691"
Sep 17 16:08:24.361601 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.361523    2581 scope.go:117] "RemoveContainer" containerID="7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458"
Sep 17 16:08:24.369748 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.369714    2581 scope.go:117] "RemoveContainer" containerID="d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae"
Sep 17 16:08:24.370094 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:08:24.370076    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae\": container with ID starting with d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae not found: ID does not exist" containerID="d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae"
Sep 17 16:08:24.370159 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.370102    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae"} err="failed to get container status \"d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae\": rpc error: code = NotFound desc = could not find container \"d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae\": container with ID starting with d8449e155dc2cf2665c8ff435b545018f047a336e519ea5f182b8a4a67c09bae not found: ID does not exist"
Sep 17 16:08:24.370159 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.370120    2581 scope.go:117] "RemoveContainer" containerID="896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691"
Sep 17 16:08:24.370376 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:08:24.370361    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691\": container with ID starting with 896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691 not found: ID does not exist" containerID="896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691"
Sep 17 16:08:24.370486 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.370380    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691"} err="failed to get container status \"896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691\": rpc error: code = NotFound desc = could not find container \"896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691\": container with ID starting with 896369e661973b79c81558551b1aae5a23c9c1e7d5c15726bfb5ef94fed52691 not found: ID does not exist"
Sep 17 16:08:24.370486 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.370400    2581 scope.go:117] "RemoveContainer" containerID="7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458"
Sep 17 16:08:24.370690 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:08:24.370669    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458\": container with ID starting with 7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458 not found: ID does not exist" containerID="7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458"
Sep 17 16:08:24.370749 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.370701    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458"} err="failed to get container status \"7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458\": rpc error: code = NotFound desc = could not find container \"7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458\": container with ID starting with 7295754a12e5b9975f0034509a485f286226ee4ce9bbc33792f07f75b30b5458 not found: ID does not exist"
Sep 17 16:08:24.371427 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.371408    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"]
Sep 17 16:08:24.374525 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:24.374504    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-fbv94"]
Sep 17 16:08:25.557690 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:25.557653    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" path="/var/lib/kubelet/pods/77855d5b-37fc-4147-9510-6c831f10bbf5/volumes"
Sep 17 16:08:26.355951 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:26.355922    2581 generic.go:356] "Generic (PLEG): container finished" podID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerID="ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296" exitCode=0
Sep 17 16:08:26.356068 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:26.355959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" event={"ID":"2dbc2636-1222-4db9-a86b-df2f974c3f64","Type":"ContainerDied","Data":"ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296"}
Sep 17 16:08:27.362436 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:27.362392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" event={"ID":"2dbc2636-1222-4db9-a86b-df2f974c3f64","Type":"ContainerStarted","Data":"6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24"}
Sep 17 16:08:27.362863 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:27.362496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" event={"ID":"2dbc2636-1222-4db9-a86b-df2f974c3f64","Type":"ContainerStarted","Data":"ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6"}
Sep 17 16:08:27.362863 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:27.362683    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:27.384487 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:27.384324    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podStartSLOduration=2.497717083 podStartE2EDuration="6.384307611s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.886590528s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:08:27.383365118 +0000 UTC m=+1798.306939684" watchObservedRunningTime="2026-09-17 16:08:27.384307611 +0000 UTC m=+1798.307882156"
Sep 17 16:08:28.367425 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:28.367392    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:28.368782 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:28.368755    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:08:29.372383 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:29.372343    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:08:29.799084 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:29.799055    2581 scope.go:117] "RemoveContainer" containerID="ff55f560472be9f737944b96b722e89b508dd324ff1cda4712ac7749d947c004"
Sep 17 16:08:29.806847 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:29.806824    2581 scope.go:117] "RemoveContainer" containerID="fcdcbfa6fa98bbd83b44274c5ce966120ecd347ccd53ef23121d6f15da4753ce"
Sep 17 16:08:34.376956 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:34.376927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:08:34.377575 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:34.377545    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:08:36.296371 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:36.296338    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"]
Sep 17 16:08:36.296820 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:36.296591    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" containerID="cri-o://316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b" gracePeriod=30
Sep 17 16:08:36.296820 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:36.296647    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kube-rbac-proxy" containerID="cri-o://6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917" gracePeriod=30
Sep 17 16:08:37.408165 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:37.408132    2581 generic.go:356] "Generic (PLEG): container finished" podID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerID="6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917" exitCode=2
Sep 17 16:08:37.408595 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:37.408207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" event={"ID":"0efe4df5-42b3-4baa-acbc-7945817ccbb3","Type":"ContainerDied","Data":"6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917"}
Sep 17 16:08:38.967668 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:38.967629    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 17 16:08:38.971995 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:38.971950    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 17 16:08:41.236774 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.236750    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:08:41.382173 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.382136    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-cabundle-cert\") pod \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") "
Sep 17 16:08:41.382350 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.382240    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kserve-provision-location\") pod \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") "
Sep 17 16:08:41.382350 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.382270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-rbac-proxy-sar-config\") pod \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") "
Sep 17 16:08:41.382350 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.382314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6nrd\" (UniqueName: \"kubernetes.io/projected/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-api-access-j6nrd\") pod \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") "
Sep 17 16:08:41.382574 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.382361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0efe4df5-42b3-4baa-acbc-7945817ccbb3-proxy-tls\") pod \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\" (UID: \"0efe4df5-42b3-4baa-acbc-7945817ccbb3\") "
Sep 17 16:08:41.382628 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.382581    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0efe4df5-42b3-4baa-acbc-7945817ccbb3" (UID: "0efe4df5-42b3-4baa-acbc-7945817ccbb3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:08:41.382675 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.382640    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0efe4df5-42b3-4baa-acbc-7945817ccbb3" (UID: "0efe4df5-42b3-4baa-acbc-7945817ccbb3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:41.382675 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.382650    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0efe4df5-42b3-4baa-acbc-7945817ccbb3" (UID: "0efe4df5-42b3-4baa-acbc-7945817ccbb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:08:41.384518 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.384491    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0efe4df5-42b3-4baa-acbc-7945817ccbb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0efe4df5-42b3-4baa-acbc-7945817ccbb3" (UID: "0efe4df5-42b3-4baa-acbc-7945817ccbb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:08:41.384624 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.384517    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-api-access-j6nrd" (OuterVolumeSpecName: "kube-api-access-j6nrd") pod "0efe4df5-42b3-4baa-acbc-7945817ccbb3" (UID: "0efe4df5-42b3-4baa-acbc-7945817ccbb3"). InnerVolumeSpecName "kube-api-access-j6nrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:08:41.428530 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.428495    2581 generic.go:356] "Generic (PLEG): container finished" podID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerID="316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b" exitCode=0
Sep 17 16:08:41.428687 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.428571    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"
Sep 17 16:08:41.428687 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.428580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" event={"ID":"0efe4df5-42b3-4baa-acbc-7945817ccbb3","Type":"ContainerDied","Data":"316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b"}
Sep 17 16:08:41.428808 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.428716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5" event={"ID":"0efe4df5-42b3-4baa-acbc-7945817ccbb3","Type":"ContainerDied","Data":"baa8909ba1a002f7061d11fd42f157d0584235c795717ca6bd61ffb434b5bd2c"}
Sep 17 16:08:41.428808 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.428764    2581 scope.go:117] "RemoveContainer" containerID="6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917"
Sep 17 16:08:41.436986 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.436970    2581 scope.go:117] "RemoveContainer" containerID="316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b"
Sep 17 16:08:41.444233 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.444213    2581 scope.go:117] "RemoveContainer" containerID="2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5"
Sep 17 16:08:41.451213 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.451195    2581 scope.go:117] "RemoveContainer" containerID="6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917"
Sep 17 16:08:41.451500 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:08:41.451480    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917\": container with ID starting with 6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917 not found: ID does not exist" containerID="6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917"
Sep 17 16:08:41.451561 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.451510    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917"} err="failed to get container status \"6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917\": rpc error: code = NotFound desc = could not find container \"6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917\": container with ID starting with 6f342fc17096ca17ce2d417a0fb0e6d042d72c050305810225d25879c2349917 not found: ID does not exist"
Sep 17 16:08:41.451561 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.451528    2581 scope.go:117] "RemoveContainer" containerID="316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b"
Sep 17 16:08:41.451774 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:08:41.451759    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b\": container with ID starting with 316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b not found: ID does not exist" containerID="316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b"
Sep 17 16:08:41.451819 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.451780    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b"} err="failed to get container status \"316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b\": rpc error: code = NotFound desc = could not find container \"316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b\": container with ID starting with 316a0d87b43dd0d1eb03e8897d07051c86cda701a9effd1354dbcebab5a9bc2b not found: ID does not exist"
Sep 17 16:08:41.451819 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.451795    2581 scope.go:117] "RemoveContainer" containerID="2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5"
Sep 17 16:08:41.452017 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:08:41.452002    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5\": container with ID starting with 2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5 not found: ID does not exist" containerID="2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5"
Sep 17 16:08:41.452068 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.452021    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5"} err="failed to get container status \"2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5\": rpc error: code = NotFound desc = could not find container \"2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5\": container with ID starting with 2184e6c0c90cace98683de0d00041e137734d868b79d8382c0572c2bbd70cec5 not found: ID does not exist"
Sep 17 16:08:41.471291 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.471268    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"]
Sep 17 16:08:41.474706 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.474686    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-5674b8c456-rd6t5"]
Sep 17 16:08:41.483811 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.483794    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6nrd\" (UniqueName: \"kubernetes.io/projected/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-api-access-j6nrd\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.483860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.483814    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0efe4df5-42b3-4baa-acbc-7945817ccbb3-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.483860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.483825    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.483860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.483834    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.483860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.483842    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0efe4df5-42b3-4baa-acbc-7945817ccbb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:08:41.557901 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.557866    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" path="/var/lib/kubelet/pods/0efe4df5-42b3-4baa-acbc-7945817ccbb3/volumes"
Sep 17 16:08:41.639700 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.639623    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"]
Sep 17 16:08:41.639949 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.639938    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container"
Sep 17 16:08:41.639996 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.639950    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container"
Sep 17 16:08:41.639996 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.639963    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kube-rbac-proxy"
Sep 17 16:08:41.639996 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.639969    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kube-rbac-proxy"
Sep 17 16:08:41.639996 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.639977    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="storage-initializer"
Sep 17 16:08:41.639996 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.639983    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="storage-initializer"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.639999    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kserve-container"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.640004    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kserve-container"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.640011    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="storage-initializer"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.640017    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="storage-initializer"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.640042    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kube-rbac-proxy"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.640048    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kube-rbac-proxy"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.640092    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kserve-container"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.640098    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kserve-container"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.640106    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efe4df5-42b3-4baa-acbc-7945817ccbb3" containerName="kube-rbac-proxy"
Sep 17 16:08:41.640171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.640112    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="77855d5b-37fc-4147-9510-6c831f10bbf5" containerName="kube-rbac-proxy"
Sep 17 16:08:41.644902 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.644886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.648347 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.648321    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 16:08:41.649130 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.649113    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 16:08:41.673755 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.673729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"]
Sep 17 16:08:41.787165 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.787128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f3646b2-cddc-423d-a08a-612f7c74feb4-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.787165 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.787166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6hpg\" (UniqueName: \"kubernetes.io/projected/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-api-access-b6hpg\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.787392 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.787256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f3646b2-cddc-423d-a08a-612f7c74feb4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.787392 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.787311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.787392 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.787383    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.888561 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.888526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.888561 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.888562    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f3646b2-cddc-423d-a08a-612f7c74feb4-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.888821 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.888581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6hpg\" (UniqueName: \"kubernetes.io/projected/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-api-access-b6hpg\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.888821 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.888627    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f3646b2-cddc-423d-a08a-612f7c74feb4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.888821 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.888659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.889073 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.889046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f3646b2-cddc-423d-a08a-612f7c74feb4-kserve-provision-location\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.889299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.889279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-cabundle-cert\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.889344 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.889294    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.891107 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.891052    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f3646b2-cddc-423d-a08a-612f7c74feb4-proxy-tls\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.910287 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.910264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6hpg\" (UniqueName: \"kubernetes.io/projected/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-api-access-b6hpg\") pod \"isvc-canary-promote-predictor-8499c568cb-ljpgg\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:41.955499 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:41.955436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:42.237267 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:42.237198    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"]
Sep 17 16:08:42.240645 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:08:42.240612    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f3646b2_cddc_423d_a08a_612f7c74feb4.slice/crio-afb3000111b3de839f0fb3cf5fbf9cbd6527008a29999647f1377718c4759b53 WatchSource:0}: Error finding container afb3000111b3de839f0fb3cf5fbf9cbd6527008a29999647f1377718c4759b53: Status 404 returned error can't find the container with id afb3000111b3de839f0fb3cf5fbf9cbd6527008a29999647f1377718c4759b53
Sep 17 16:08:42.437488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:42.435067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" event={"ID":"6f3646b2-cddc-423d-a08a-612f7c74feb4","Type":"ContainerStarted","Data":"8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa"}
Sep 17 16:08:42.437488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:42.435143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" event={"ID":"6f3646b2-cddc-423d-a08a-612f7c74feb4","Type":"ContainerStarted","Data":"afb3000111b3de839f0fb3cf5fbf9cbd6527008a29999647f1377718c4759b53"}
Sep 17 16:08:44.378530 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:44.378488    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:08:46.458025 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:46.457991    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerID="8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa" exitCode=0
Sep 17 16:08:46.458387 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:46.458046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" event={"ID":"6f3646b2-cddc-423d-a08a-612f7c74feb4","Type":"ContainerDied","Data":"8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa"}
Sep 17 16:08:47.464293 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:47.464256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" event={"ID":"6f3646b2-cddc-423d-a08a-612f7c74feb4","Type":"ContainerStarted","Data":"7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef"}
Sep 17 16:08:47.464780 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:47.464330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" event={"ID":"6f3646b2-cddc-423d-a08a-612f7c74feb4","Type":"ContainerStarted","Data":"5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef"}
Sep 17 16:08:47.464780 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:47.464565    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:47.498686 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:47.498637    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podStartSLOduration=2.806668976 podStartE2EDuration="6.498620528s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.691951552s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:08:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:08:47.494555657 +0000 UTC m=+1818.418130224" watchObservedRunningTime="2026-09-17 16:08:47.498620528 +0000 UTC m=+1818.422195094"
Sep 17 16:08:48.469022 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:48.468990    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:48.470509 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:48.470479    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:08:49.473579 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:49.473537    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:08:54.378022 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:54.377980    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:08:54.478352 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:54.478322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:08:54.478821 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:08:54.478797    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:09:04.378339 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:04.378295    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:09:04.479503 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:04.479451    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:09:14.378308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:14.378269    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:09:14.479156 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:14.479115    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:09:24.377897 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:24.377854    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:09:24.479797 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:24.479755    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:09:34.378453 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:34.378406    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:09:34.478794 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:34.478756    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:09:40.552955 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:40.552908    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 17 16:09:44.479042 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:44.479001    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:09:50.553631 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:50.553597    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:09:51.282029 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:51.281999    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"]
Sep 17 16:09:51.282242 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:51.282220    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container" containerID="cri-o://ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6" gracePeriod=30
Sep 17 16:09:51.282304 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:51.282260    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kube-rbac-proxy" containerID="cri-o://6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24" gracePeriod=30
Sep 17 16:09:51.738825 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:51.738792    2581 generic.go:356] "Generic (PLEG): container finished" podID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerID="6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24" exitCode=2
Sep 17 16:09:51.739236 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:51.738864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" event={"ID":"2dbc2636-1222-4db9-a86b-df2f974c3f64","Type":"ContainerDied","Data":"6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24"}
Sep 17 16:09:54.373417 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:54.373374    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Sep 17 16:09:54.479484 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:54.479430    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 17 16:09:55.913674 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:55.913651    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:09:56.029642 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.029560    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-cabundle-cert\") pod \"2dbc2636-1222-4db9-a86b-df2f974c3f64\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") "
Sep 17 16:09:56.029642 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.029612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-rbac-proxy-sar-config\") pod \"2dbc2636-1222-4db9-a86b-df2f974c3f64\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") "
Sep 17 16:09:56.029642 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.029638    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxn8n\" (UniqueName: \"kubernetes.io/projected/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-api-access-dxn8n\") pod \"2dbc2636-1222-4db9-a86b-df2f974c3f64\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") "
Sep 17 16:09:56.029933 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.029705    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbc2636-1222-4db9-a86b-df2f974c3f64-proxy-tls\") pod \"2dbc2636-1222-4db9-a86b-df2f974c3f64\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") "
Sep 17 16:09:56.029933 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.029723    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dbc2636-1222-4db9-a86b-df2f974c3f64-kserve-provision-location\") pod \"2dbc2636-1222-4db9-a86b-df2f974c3f64\" (UID: \"2dbc2636-1222-4db9-a86b-df2f974c3f64\") "
Sep 17 16:09:56.030063 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.030042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dbc2636-1222-4db9-a86b-df2f974c3f64-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2dbc2636-1222-4db9-a86b-df2f974c3f64" (UID: "2dbc2636-1222-4db9-a86b-df2f974c3f64"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:09:56.030063 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.030048    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2dbc2636-1222-4db9-a86b-df2f974c3f64" (UID: "2dbc2636-1222-4db9-a86b-df2f974c3f64"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:09:56.030144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.030042    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2dbc2636-1222-4db9-a86b-df2f974c3f64" (UID: "2dbc2636-1222-4db9-a86b-df2f974c3f64"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:09:56.031842 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.031819    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dbc2636-1222-4db9-a86b-df2f974c3f64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2dbc2636-1222-4db9-a86b-df2f974c3f64" (UID: "2dbc2636-1222-4db9-a86b-df2f974c3f64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:09:56.031842 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.031833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-api-access-dxn8n" (OuterVolumeSpecName: "kube-api-access-dxn8n") pod "2dbc2636-1222-4db9-a86b-df2f974c3f64" (UID: "2dbc2636-1222-4db9-a86b-df2f974c3f64"). InnerVolumeSpecName "kube-api-access-dxn8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:09:56.130421 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.130380    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxn8n\" (UniqueName: \"kubernetes.io/projected/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-api-access-dxn8n\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:09:56.130421 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.130415    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2dbc2636-1222-4db9-a86b-df2f974c3f64-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:09:56.130421 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.130429    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2dbc2636-1222-4db9-a86b-df2f974c3f64-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:09:56.130671 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.130443    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:09:56.130671 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.130455    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2dbc2636-1222-4db9-a86b-df2f974c3f64-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:09:56.764428 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.764395    2581 generic.go:356] "Generic (PLEG): container finished" podID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerID="ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6" exitCode=0
Sep 17 16:09:56.764605 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.764449    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"
Sep 17 16:09:56.764605 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.764491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" event={"ID":"2dbc2636-1222-4db9-a86b-df2f974c3f64","Type":"ContainerDied","Data":"ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6"}
Sep 17 16:09:56.764605 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.764552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr" event={"ID":"2dbc2636-1222-4db9-a86b-df2f974c3f64","Type":"ContainerDied","Data":"9b01ebb2c97095b10d1bb1827e4c4e179bd99824def4df1f6af42c4f50e128f4"}
Sep 17 16:09:56.764605 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.764578    2581 scope.go:117] "RemoveContainer" containerID="6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24"
Sep 17 16:09:56.772761 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.772741    2581 scope.go:117] "RemoveContainer" containerID="ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6"
Sep 17 16:09:56.779660 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.779644    2581 scope.go:117] "RemoveContainer" containerID="ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296"
Sep 17 16:09:56.786763 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.786746    2581 scope.go:117] "RemoveContainer" containerID="6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24"
Sep 17 16:09:56.787013 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:09:56.786994    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24\": container with ID starting with 6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24 not found: ID does not exist" containerID="6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24"
Sep 17 16:09:56.787063 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.787023    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24"} err="failed to get container status \"6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24\": rpc error: code = NotFound desc = could not find container \"6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24\": container with ID starting with 6fcbb17b0a903f529ef8403a4d0504a72ed599e39154427f9fdea571685daa24 not found: ID does not exist"
Sep 17 16:09:56.787063 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.787039    2581 scope.go:117] "RemoveContainer" containerID="ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6"
Sep 17 16:09:56.787272 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:09:56.787253    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6\": container with ID starting with ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6 not found: ID does not exist" containerID="ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6"
Sep 17 16:09:56.787338 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.787281    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6"} err="failed to get container status \"ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6\": rpc error: code = NotFound desc = could not find container \"ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6\": container with ID starting with ea89bf392a30b84e53a23e2a6f736f3fee0910846505222ca8c736f9f1c273d6 not found: ID does not exist"
Sep 17 16:09:56.787338 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.787319    2581 scope.go:117] "RemoveContainer" containerID="ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296"
Sep 17 16:09:56.787573 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:09:56.787557    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296\": container with ID starting with ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296 not found: ID does not exist" containerID="ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296"
Sep 17 16:09:56.787622 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.787577    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296"} err="failed to get container status \"ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296\": rpc error: code = NotFound desc = could not find container \"ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296\": container with ID starting with ed88ce8e8c331a94a15afd556bc628aa1a3d3e2c0108b41d28724fec5a8c5296 not found: ID does not exist"
Sep 17 16:09:56.799420 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.799399    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"]
Sep 17 16:09:56.804318 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:56.804297    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-68f4f4d797-c6nvr"]
Sep 17 16:09:57.557705 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:09:57.557672    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" path="/var/lib/kubelet/pods/2dbc2636-1222-4db9-a86b-df2f974c3f64/volumes"
Sep 17 16:10:04.479611 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:04.479581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:10:04.865162 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:04.860691    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"]
Sep 17 16:10:04.865162 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:04.864725    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container" containerID="cri-o://5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef" gracePeriod=30
Sep 17 16:10:04.865162 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:04.865009    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kube-rbac-proxy" containerID="cri-o://7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef" gracePeriod=30
Sep 17 16:10:05.804488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:05.804428    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerID="7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef" exitCode=2
Sep 17 16:10:05.804941 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:05.804488    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" event={"ID":"6f3646b2-cddc-423d-a08a-612f7c74feb4","Type":"ContainerDied","Data":"7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef"}
Sep 17 16:10:09.500977 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.500955    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:10:09.650784 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.650703    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-rbac-proxy-sar-config\") pod \"6f3646b2-cddc-423d-a08a-612f7c74feb4\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") "
Sep 17 16:10:09.650784 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.650743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f3646b2-cddc-423d-a08a-612f7c74feb4-kserve-provision-location\") pod \"6f3646b2-cddc-423d-a08a-612f7c74feb4\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") "
Sep 17 16:10:09.650784 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.650774    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f3646b2-cddc-423d-a08a-612f7c74feb4-proxy-tls\") pod \"6f3646b2-cddc-423d-a08a-612f7c74feb4\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") "
Sep 17 16:10:09.651090 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.650838    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6hpg\" (UniqueName: \"kubernetes.io/projected/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-api-access-b6hpg\") pod \"6f3646b2-cddc-423d-a08a-612f7c74feb4\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") "
Sep 17 16:10:09.651090 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.650853    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-cabundle-cert\") pod \"6f3646b2-cddc-423d-a08a-612f7c74feb4\" (UID: \"6f3646b2-cddc-423d-a08a-612f7c74feb4\") "
Sep 17 16:10:09.651198 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.651070    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f3646b2-cddc-423d-a08a-612f7c74feb4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f3646b2-cddc-423d-a08a-612f7c74feb4" (UID: "6f3646b2-cddc-423d-a08a-612f7c74feb4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:10:09.651246 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.651212    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f3646b2-cddc-423d-a08a-612f7c74feb4" (UID: "6f3646b2-cddc-423d-a08a-612f7c74feb4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:09.651246 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.651206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f3646b2-cddc-423d-a08a-612f7c74feb4" (UID: "6f3646b2-cddc-423d-a08a-612f7c74feb4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:10:09.652986 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.652957    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f3646b2-cddc-423d-a08a-612f7c74feb4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f3646b2-cddc-423d-a08a-612f7c74feb4" (UID: "6f3646b2-cddc-423d-a08a-612f7c74feb4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:10:09.653096 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.652988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-api-access-b6hpg" (OuterVolumeSpecName: "kube-api-access-b6hpg") pod "6f3646b2-cddc-423d-a08a-612f7c74feb4" (UID: "6f3646b2-cddc-423d-a08a-612f7c74feb4"). InnerVolumeSpecName "kube-api-access-b6hpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:10:09.751659 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.751630    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f3646b2-cddc-423d-a08a-612f7c74feb4-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.751659 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.751654    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6hpg\" (UniqueName: \"kubernetes.io/projected/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-api-access-b6hpg\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.751659 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.751664    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.751875 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.751672    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f3646b2-cddc-423d-a08a-612f7c74feb4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.751875 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.751682    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f3646b2-cddc-423d-a08a-612f7c74feb4-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:10:09.822783 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.822749    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerID="5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef" exitCode=0
Sep 17 16:10:09.822930 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.822803    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"
Sep 17 16:10:09.822930 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.822822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" event={"ID":"6f3646b2-cddc-423d-a08a-612f7c74feb4","Type":"ContainerDied","Data":"5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef"}
Sep 17 16:10:09.822930 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.822868    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" event={"ID":"6f3646b2-cddc-423d-a08a-612f7c74feb4","Type":"ContainerDied","Data":"afb3000111b3de839f0fb3cf5fbf9cbd6527008a29999647f1377718c4759b53"}
Sep 17 16:10:09.822930 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.822893    2581 scope.go:117] "RemoveContainer" containerID="7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef"
Sep 17 16:10:09.831440 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.831420    2581 scope.go:117] "RemoveContainer" containerID="5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef"
Sep 17 16:10:09.838385 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.838367    2581 scope.go:117] "RemoveContainer" containerID="8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa"
Sep 17 16:10:09.845483 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.845406    2581 scope.go:117] "RemoveContainer" containerID="7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef"
Sep 17 16:10:09.845871 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:10:09.845852    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef\": container with ID starting with 7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef not found: ID does not exist" containerID="7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef"
Sep 17 16:10:09.845932 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.845895    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef"} err="failed to get container status \"7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef\": rpc error: code = NotFound desc = could not find container \"7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef\": container with ID starting with 7b6d7c16fdbbc8ae711f44f80a34ee78e117b9c9c688e42415954ce20b34bbef not found: ID does not exist"
Sep 17 16:10:09.845932 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.845912    2581 scope.go:117] "RemoveContainer" containerID="5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef"
Sep 17 16:10:09.846177 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:10:09.846161    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef\": container with ID starting with 5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef not found: ID does not exist" containerID="5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef"
Sep 17 16:10:09.846223 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.846181    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef"} err="failed to get container status \"5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef\": rpc error: code = NotFound desc = could not find container \"5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef\": container with ID starting with 5c80965a564475398a3eb63d864416890aedcef19b37bc50c274100a9f1eb4ef not found: ID does not exist"
Sep 17 16:10:09.846223 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.846196    2581 scope.go:117] "RemoveContainer" containerID="8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa"
Sep 17 16:10:09.846531 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:10:09.846509    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa\": container with ID starting with 8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa not found: ID does not exist" containerID="8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa"
Sep 17 16:10:09.846579 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.846540    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa"} err="failed to get container status \"8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa\": rpc error: code = NotFound desc = could not find container \"8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa\": container with ID starting with 8b82c8400ce028117413c0a268e75f7959b2d5580129b07397dc09ab325b63fa not found: ID does not exist"
Sep 17 16:10:09.847273 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.847257    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"]
Sep 17 16:10:09.853173 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:09.853154    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg"]
Sep 17 16:10:10.474929 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:10.474889    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-8499c568cb-ljpgg" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 17 16:10:11.557154 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.557123    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" path="/var/lib/kubelet/pods/6f3646b2-cddc-423d-a08a-612f7c74feb4/volumes"
Sep 17 16:10:11.572820 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.572793    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"]
Sep 17 16:10:11.573105 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573093    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="storage-initializer"
Sep 17 16:10:11.573145 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573107    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="storage-initializer"
Sep 17 16:10:11.573145 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573122    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="storage-initializer"
Sep 17 16:10:11.573145 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573127    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="storage-initializer"
Sep 17 16:10:11.573145 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573136    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kube-rbac-proxy"
Sep 17 16:10:11.573145 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573142    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kube-rbac-proxy"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573154    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kube-rbac-proxy"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573159    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kube-rbac-proxy"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573167    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573172    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573179    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573184    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573229    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kserve-container"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573236    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dbc2636-1222-4db9-a86b-df2f974c3f64" containerName="kube-rbac-proxy"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573242    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kube-rbac-proxy"
Sep 17 16:10:11.573299 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.573250    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f3646b2-cddc-423d-a08a-612f7c74feb4" containerName="kserve-container"
Sep 17 16:10:11.577890 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.577873    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.582170 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.582144    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:10:11.582284 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.582195    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 16:10:11.583165 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.583143    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:10:11.583516 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.583500    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 16:10:11.583516 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.583507    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:10:11.583654 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.583524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:10:11.583654 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.583511    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:10:11.589172 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.589152    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"]
Sep 17 16:10:11.665941 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.665909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2j6v\" (UniqueName: \"kubernetes.io/projected/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-api-access-f2j6v\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.666086 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.665946    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.666086 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.665974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f670c1e-d0a0-4c01-b093-cf0795418d22-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.666178 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.666135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f670c1e-d0a0-4c01-b093-cf0795418d22-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.666178 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.666159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.767236 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.767208    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f670c1e-d0a0-4c01-b093-cf0795418d22-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.767374 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.767241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.767374 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.767293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2j6v\" (UniqueName: \"kubernetes.io/projected/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-api-access-f2j6v\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.767374 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.767318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.767374 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.767346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f670c1e-d0a0-4c01-b093-cf0795418d22-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.767800 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.767776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f670c1e-d0a0-4c01-b093-cf0795418d22-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.767984 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.767964    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.768049 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.767987    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.769649 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.769627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f670c1e-d0a0-4c01-b093-cf0795418d22-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.778809 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.778787    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2j6v\" (UniqueName: \"kubernetes.io/projected/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-api-access-f2j6v\") pod \"isvc-canary-promote-v2-predictor-5654b4fcff-rq467\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:11.888352 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:11.888273    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:12.165311 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:12.165288    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"]
Sep 17 16:10:12.167401 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:10:12.167374    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f670c1e_d0a0_4c01_b093_cf0795418d22.slice/crio-643dcdf7be80a12952cd7795d875b02c3277857bdda2d74331d584627c4b8bc7 WatchSource:0}: Error finding container 643dcdf7be80a12952cd7795d875b02c3277857bdda2d74331d584627c4b8bc7: Status 404 returned error can't find the container with id 643dcdf7be80a12952cd7795d875b02c3277857bdda2d74331d584627c4b8bc7
Sep 17 16:10:12.836370 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:12.836330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" event={"ID":"6f670c1e-d0a0-4c01-b093-cf0795418d22","Type":"ContainerStarted","Data":"2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1"}
Sep 17 16:10:12.836871 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:12.836388    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" event={"ID":"6f670c1e-d0a0-4c01-b093-cf0795418d22","Type":"ContainerStarted","Data":"643dcdf7be80a12952cd7795d875b02c3277857bdda2d74331d584627c4b8bc7"}
Sep 17 16:10:16.854276 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:16.854238    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerID="2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1" exitCode=0
Sep 17 16:10:16.854685 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:16.854324    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" event={"ID":"6f670c1e-d0a0-4c01-b093-cf0795418d22","Type":"ContainerDied","Data":"2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1"}
Sep 17 16:10:17.860383 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:17.860343    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" event={"ID":"6f670c1e-d0a0-4c01-b093-cf0795418d22","Type":"ContainerStarted","Data":"8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e"}
Sep 17 16:10:17.860791 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:17.860402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" event={"ID":"6f670c1e-d0a0-4c01-b093-cf0795418d22","Type":"ContainerStarted","Data":"ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9"}
Sep 17 16:10:17.860791 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:17.860752    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:17.860903 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:17.860887    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:17.862290 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:17.862264    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:10:17.891294 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:17.891247    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podStartSLOduration=3.193514446 podStartE2EDuration="6.891233133s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.697718687s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:10:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:10:17.889712839 +0000 UTC m=+1908.813287417" watchObservedRunningTime="2026-09-17 16:10:17.891233133 +0000 UTC m=+1908.814807701"
Sep 17 16:10:18.864594 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:18.864558    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:10:19.868870 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:19.868832    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:10:24.873171 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:24.873141    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:10:24.873652 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:24.873624    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:10:34.874143 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:34.874105    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:10:44.873762 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:44.873722    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:10:54.873758 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:10:54.873719    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:11:04.874399 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:04.874356    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:11:14.874361 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:14.874312    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:11:24.873805 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:24.873720    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:11:34.874446 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:34.874414    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:11:41.723938 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:41.723906    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"]
Sep 17 16:11:41.724330 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:41.724132    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" containerID="cri-o://ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9" gracePeriod=30
Sep 17 16:11:41.724330 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:41.724167    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kube-rbac-proxy" containerID="cri-o://8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e" gracePeriod=30
Sep 17 16:11:42.216157 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:42.216118    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerID="8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e" exitCode=2
Sep 17 16:11:42.216350 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:42.216201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" event={"ID":"6f670c1e-d0a0-4c01-b093-cf0795418d22","Type":"ContainerDied","Data":"8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e"}
Sep 17 16:11:44.870110 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:44.870062    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 17 16:11:44.874416 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:44.874390    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 17 16:11:46.558865 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.558841    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:11:46.681445 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.681367    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-rbac-proxy-sar-config\") pod \"6f670c1e-d0a0-4c01-b093-cf0795418d22\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") "
Sep 17 16:11:46.681445 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.681404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-cabundle-cert\") pod \"6f670c1e-d0a0-4c01-b093-cf0795418d22\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") "
Sep 17 16:11:46.681639 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.681516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2j6v\" (UniqueName: \"kubernetes.io/projected/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-api-access-f2j6v\") pod \"6f670c1e-d0a0-4c01-b093-cf0795418d22\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") "
Sep 17 16:11:46.681639 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.681581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f670c1e-d0a0-4c01-b093-cf0795418d22-kserve-provision-location\") pod \"6f670c1e-d0a0-4c01-b093-cf0795418d22\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") "
Sep 17 16:11:46.681639 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.681613    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f670c1e-d0a0-4c01-b093-cf0795418d22-proxy-tls\") pod \"6f670c1e-d0a0-4c01-b093-cf0795418d22\" (UID: \"6f670c1e-d0a0-4c01-b093-cf0795418d22\") "
Sep 17 16:11:46.681843 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.681816    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f670c1e-d0a0-4c01-b093-cf0795418d22" (UID: "6f670c1e-d0a0-4c01-b093-cf0795418d22"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:46.681915 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.681795    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f670c1e-d0a0-4c01-b093-cf0795418d22" (UID: "6f670c1e-d0a0-4c01-b093-cf0795418d22"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:11:46.681915 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.681897    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f670c1e-d0a0-4c01-b093-cf0795418d22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f670c1e-d0a0-4c01-b093-cf0795418d22" (UID: "6f670c1e-d0a0-4c01-b093-cf0795418d22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:11:46.683410 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.683392    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-api-access-f2j6v" (OuterVolumeSpecName: "kube-api-access-f2j6v") pod "6f670c1e-d0a0-4c01-b093-cf0795418d22" (UID: "6f670c1e-d0a0-4c01-b093-cf0795418d22"). InnerVolumeSpecName "kube-api-access-f2j6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:11:46.683580 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.683560    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f670c1e-d0a0-4c01-b093-cf0795418d22-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f670c1e-d0a0-4c01-b093-cf0795418d22" (UID: "6f670c1e-d0a0-4c01-b093-cf0795418d22"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:11:46.782215 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.782179    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f670c1e-d0a0-4c01-b093-cf0795418d22-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:11:46.782215 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.782212    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f670c1e-d0a0-4c01-b093-cf0795418d22-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:11:46.782215 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.782223    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:11:46.782479 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.782233    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f670c1e-d0a0-4c01-b093-cf0795418d22-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:11:46.782479 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:46.782242    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2j6v\" (UniqueName: \"kubernetes.io/projected/6f670c1e-d0a0-4c01-b093-cf0795418d22-kube-api-access-f2j6v\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:11:47.174231 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.174192    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"]
Sep 17 16:11:47.174664 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.174647    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="storage-initializer"
Sep 17 16:11:47.174727 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.174666    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="storage-initializer"
Sep 17 16:11:47.174727 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.174683    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kube-rbac-proxy"
Sep 17 16:11:47.174727 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.174692    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kube-rbac-proxy"
Sep 17 16:11:47.174727 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.174706    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container"
Sep 17 16:11:47.174727 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.174714    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container"
Sep 17 16:11:47.174904 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.174781    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kserve-container"
Sep 17 16:11:47.174904 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.174795    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerName="kube-rbac-proxy"
Sep 17 16:11:47.178178 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.178160    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.182071 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.181900    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 16:11:47.182247 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.182228    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 16:11:47.237658 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.237625    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f670c1e-d0a0-4c01-b093-cf0795418d22" containerID="ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9" exitCode=0
Sep 17 16:11:47.237865 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.237700    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"
Sep 17 16:11:47.237865 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.237712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" event={"ID":"6f670c1e-d0a0-4c01-b093-cf0795418d22","Type":"ContainerDied","Data":"ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9"}
Sep 17 16:11:47.237865 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.237768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467" event={"ID":"6f670c1e-d0a0-4c01-b093-cf0795418d22","Type":"ContainerDied","Data":"643dcdf7be80a12952cd7795d875b02c3277857bdda2d74331d584627c4b8bc7"}
Sep 17 16:11:47.237865 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.237795    2581 scope.go:117] "RemoveContainer" containerID="8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e"
Sep 17 16:11:47.247011 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.246994    2581 scope.go:117] "RemoveContainer" containerID="ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9"
Sep 17 16:11:47.247658 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.247632    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"]
Sep 17 16:11:47.255195 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.255178    2581 scope.go:117] "RemoveContainer" containerID="2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1"
Sep 17 16:11:47.262755 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.262738    2581 scope.go:117] "RemoveContainer" containerID="8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e"
Sep 17 16:11:47.263062 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:11:47.263042    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e\": container with ID starting with 8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e not found: ID does not exist" containerID="8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e"
Sep 17 16:11:47.263118 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.263072    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e"} err="failed to get container status \"8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e\": rpc error: code = NotFound desc = could not find container \"8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e\": container with ID starting with 8755febaa4a1f55ec7d8f617470f93ab8b0e3018790de06e3291ddb863a4dc6e not found: ID does not exist"
Sep 17 16:11:47.263118 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.263091    2581 scope.go:117] "RemoveContainer" containerID="ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9"
Sep 17 16:11:47.263350 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:11:47.263328    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9\": container with ID starting with ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9 not found: ID does not exist" containerID="ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9"
Sep 17 16:11:47.263441 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.263354    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9"} err="failed to get container status \"ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9\": rpc error: code = NotFound desc = could not find container \"ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9\": container with ID starting with ce1dca9526a16fc2c525602b38e1fe44be04fde5c2ecffc18b5ab805bacfe3d9 not found: ID does not exist"
Sep 17 16:11:47.263441 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.263376    2581 scope.go:117] "RemoveContainer" containerID="2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1"
Sep 17 16:11:47.263640 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:11:47.263623    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1\": container with ID starting with 2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1 not found: ID does not exist" containerID="2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1"
Sep 17 16:11:47.263683 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.263645    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1"} err="failed to get container status \"2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1\": rpc error: code = NotFound desc = could not find container \"2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1\": container with ID starting with 2e6224ce243e74d7e971d0aac0f4c6deb10c00a4a8f712a2f86cb1babc8e79d1 not found: ID does not exist"
Sep 17 16:11:47.267865 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.267843    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"]
Sep 17 16:11:47.270201 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.270183    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-5654b4fcff-rq467"]
Sep 17 16:11:47.286137 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.286116    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.286235 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.286144    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.286274 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.286244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff0e821e-7f9c-4b2e-afba-c2fed0704091-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.286311 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.286281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.286352 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.286312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fvwr\" (UniqueName: \"kubernetes.io/projected/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-api-access-2fvwr\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.387502 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.387438    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff0e821e-7f9c-4b2e-afba-c2fed0704091-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.387690 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.387522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.387690 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.387551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fvwr\" (UniqueName: \"kubernetes.io/projected/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-api-access-2fvwr\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.387690 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.387589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.387690 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.387614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.388128 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.388099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.388414 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.388393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-cabundle-cert\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.388459 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.388393    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.389929 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.389912    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff0e821e-7f9c-4b2e-afba-c2fed0704091-proxy-tls\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.401471 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.401438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fvwr\" (UniqueName: \"kubernetes.io/projected/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-api-access-2fvwr\") pod \"isvc-canary-rollback-predictor-78f9bdc99b-kvk8k\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.488504 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.488360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:47.560168 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.560137    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f670c1e-d0a0-4c01-b093-cf0795418d22" path="/var/lib/kubelet/pods/6f670c1e-d0a0-4c01-b093-cf0795418d22/volumes"
Sep 17 16:11:47.773260 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:11:47.773230    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0e821e_7f9c_4b2e_afba_c2fed0704091.slice/crio-49244796c0264bf9c5ca5e20529ab876a0cda6f71c9d168b7b47000e3ff3dff5 WatchSource:0}: Error finding container 49244796c0264bf9c5ca5e20529ab876a0cda6f71c9d168b7b47000e3ff3dff5: Status 404 returned error can't find the container with id 49244796c0264bf9c5ca5e20529ab876a0cda6f71c9d168b7b47000e3ff3dff5
Sep 17 16:11:47.773375 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:47.773296    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"]
Sep 17 16:11:48.244385 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:48.244347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" event={"ID":"ff0e821e-7f9c-4b2e-afba-c2fed0704091","Type":"ContainerStarted","Data":"145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389"}
Sep 17 16:11:48.244576 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:48.244449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" event={"ID":"ff0e821e-7f9c-4b2e-afba-c2fed0704091","Type":"ContainerStarted","Data":"49244796c0264bf9c5ca5e20529ab876a0cda6f71c9d168b7b47000e3ff3dff5"}
Sep 17 16:11:52.264344 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:52.264311    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerID="145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389" exitCode=0
Sep 17 16:11:52.264737 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:52.264382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" event={"ID":"ff0e821e-7f9c-4b2e-afba-c2fed0704091","Type":"ContainerDied","Data":"145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389"}
Sep 17 16:11:53.270479 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:53.270441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" event={"ID":"ff0e821e-7f9c-4b2e-afba-c2fed0704091","Type":"ContainerStarted","Data":"d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a"}
Sep 17 16:11:53.270876 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:53.270521    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" event={"ID":"ff0e821e-7f9c-4b2e-afba-c2fed0704091","Type":"ContainerStarted","Data":"2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a"}
Sep 17 16:11:53.270876 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:53.270800    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:53.270977 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:53.270957    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:53.272260 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:53.272232    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:11:53.292535 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:53.292484    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podStartSLOduration=2.69498246 podStartE2EDuration="6.292443714s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.597461254s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:11:53.291452956 +0000 UTC m=+2004.215027523" watchObservedRunningTime="2026-09-17 16:11:53.292443714 +0000 UTC m=+2004.216018278"
Sep 17 16:11:54.274908 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:54.274864    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:11:59.279693 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:59.279665    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:11:59.280179 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:11:59.280121    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:12:09.280141 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:09.280100    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:12:19.280960 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.280917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:12:19.699034 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.698999    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"]
Sep 17 16:12:19.702702 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.702680    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.704776 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.704756    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:12:19.704863 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.704773    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 17 16:12:19.758457 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.758421    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"]
Sep 17 16:12:19.768864 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.768837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.769010 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.768879    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba4e01-c585-43cb-955a-ce6750ec1323-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.769010 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.768935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9n24\" (UniqueName: \"kubernetes.io/projected/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-api-access-j9n24\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.769010 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.768961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.769010 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.768988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eba4e01-c585-43cb-955a-ce6750ec1323-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.870014 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.869975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9n24\" (UniqueName: \"kubernetes.io/projected/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-api-access-j9n24\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.870188 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.870024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.870188 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.870057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eba4e01-c585-43cb-955a-ce6750ec1323-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.870188 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.870122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.876074 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.870161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba4e01-c585-43cb-955a-ce6750ec1323-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.876074 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.870748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba4e01-c585-43cb-955a-ce6750ec1323-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.876074 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.871251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.876074 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.871376    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.877935 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.877911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eba4e01-c585-43cb-955a-ce6750ec1323-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:19.879553 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:19.879532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9n24\" (UniqueName: \"kubernetes.io/projected/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-api-access-j9n24\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:20.012541 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:20.012420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:20.292740 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:20.292706    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"]
Sep 17 16:12:20.296047 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:12:20.296009    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eba4e01_c585_43cb_955a_ce6750ec1323.slice/crio-08804ad17e136a7368c3e00efb16972a66abf7d0d7b90efa434eebddf4df1152 WatchSource:0}: Error finding container 08804ad17e136a7368c3e00efb16972a66abf7d0d7b90efa434eebddf4df1152: Status 404 returned error can't find the container with id 08804ad17e136a7368c3e00efb16972a66abf7d0d7b90efa434eebddf4df1152
Sep 17 16:12:20.386280 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:20.386247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" event={"ID":"5eba4e01-c585-43cb-955a-ce6750ec1323","Type":"ContainerStarted","Data":"974cef0898c1c667ba4aed37a194108177a19befb59d5fac9cc3a0571f734b18"}
Sep 17 16:12:20.386507 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:20.386304    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" event={"ID":"5eba4e01-c585-43cb-955a-ce6750ec1323","Type":"ContainerStarted","Data":"08804ad17e136a7368c3e00efb16972a66abf7d0d7b90efa434eebddf4df1152"}
Sep 17 16:12:25.408535 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:25.408498    2581 generic.go:356] "Generic (PLEG): container finished" podID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerID="974cef0898c1c667ba4aed37a194108177a19befb59d5fac9cc3a0571f734b18" exitCode=0
Sep 17 16:12:25.408953 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:25.408578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" event={"ID":"5eba4e01-c585-43cb-955a-ce6750ec1323","Type":"ContainerDied","Data":"974cef0898c1c667ba4aed37a194108177a19befb59d5fac9cc3a0571f734b18"}
Sep 17 16:12:25.409876 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:25.409862    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:12:29.280925 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:29.280834    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:12:29.429685 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:29.429646    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" event={"ID":"5eba4e01-c585-43cb-955a-ce6750ec1323","Type":"ContainerStarted","Data":"815a71e7076c707bf6fa43ea2f40f9bc67ed3b60a5b60b6a7514be2bdb8a7249"}
Sep 17 16:12:29.429858 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:29.429709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" event={"ID":"5eba4e01-c585-43cb-955a-ce6750ec1323","Type":"ContainerStarted","Data":"00756ea00ee2a31355118619cbcd88dc4b2d6919e3bae0911eb64c85ad7895cf"}
Sep 17 16:12:29.430008 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:29.429989    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:29.430157 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:29.430141    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:29.431533 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:29.431504    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:12:29.476199 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:29.476141    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podStartSLOduration=2.046402404 podStartE2EDuration="10.476122597s" totalImagesPullingTime="3.591962237s" totalInitContainerRuntime="4.837757956s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:12:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:12:29.474027234 +0000 UTC m=+2040.397601802" watchObservedRunningTime="2026-09-17 16:12:29.476122597 +0000 UTC m=+2040.399697168"
Sep 17 16:12:30.433904 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:30.433866    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:12:35.438622 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:35.438589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:35.439083 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:35.439056    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 17 16:12:39.280590 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:39.280542    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:12:45.439995 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:45.439964    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:12:49.280915 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:49.280872    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:12:59.280848 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:12:59.280762    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:13:01.200480 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:01.200432    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"]
Sep 17 16:13:01.200870 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:01.200791    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kserve-container" containerID="cri-o://00756ea00ee2a31355118619cbcd88dc4b2d6919e3bae0911eb64c85ad7895cf" gracePeriod=30
Sep 17 16:13:01.200933 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:01.200860    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy" containerID="cri-o://815a71e7076c707bf6fa43ea2f40f9bc67ed3b60a5b60b6a7514be2bdb8a7249" gracePeriod=30
Sep 17 16:13:01.570226 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:01.570188    2581 generic.go:356] "Generic (PLEG): container finished" podID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerID="815a71e7076c707bf6fa43ea2f40f9bc67ed3b60a5b60b6a7514be2bdb8a7249" exitCode=2
Sep 17 16:13:01.570402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:01.570258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" event={"ID":"5eba4e01-c585-43cb-955a-ce6750ec1323","Type":"ContainerDied","Data":"815a71e7076c707bf6fa43ea2f40f9bc67ed3b60a5b60b6a7514be2bdb8a7249"}
Sep 17 16:13:04.552965 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:04.552915    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 17 16:13:05.434796 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:05.434752    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 16:13:10.434964 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:10.434917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 16:13:14.554377 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:14.554340    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:13:15.434990 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:15.434939    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 16:13:15.435178 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:15.435093    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:13:20.434455 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:20.434410    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 16:13:25.434640 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:25.434589    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 16:13:30.434952 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:30.434910    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 17 16:13:31.698458 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.698424    2581 generic.go:356] "Generic (PLEG): container finished" podID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerID="00756ea00ee2a31355118619cbcd88dc4b2d6919e3bae0911eb64c85ad7895cf" exitCode=137
Sep 17 16:13:31.698857 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.698497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" event={"ID":"5eba4e01-c585-43cb-955a-ce6750ec1323","Type":"ContainerDied","Data":"00756ea00ee2a31355118619cbcd88dc4b2d6919e3bae0911eb64c85ad7895cf"}
Sep 17 16:13:31.848340 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.848313    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:13:31.898298 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.898260    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-rbac-proxy-sar-config\") pod \"5eba4e01-c585-43cb-955a-ce6750ec1323\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") "
Sep 17 16:13:31.898298 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.898304    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-cabundle-cert\") pod \"5eba4e01-c585-43cb-955a-ce6750ec1323\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") "
Sep 17 16:13:31.898613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.898335    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eba4e01-c585-43cb-955a-ce6750ec1323-proxy-tls\") pod \"5eba4e01-c585-43cb-955a-ce6750ec1323\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") "
Sep 17 16:13:31.898613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.898374    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9n24\" (UniqueName: \"kubernetes.io/projected/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-api-access-j9n24\") pod \"5eba4e01-c585-43cb-955a-ce6750ec1323\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") "
Sep 17 16:13:31.898613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.898420    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba4e01-c585-43cb-955a-ce6750ec1323-kserve-provision-location\") pod \"5eba4e01-c585-43cb-955a-ce6750ec1323\" (UID: \"5eba4e01-c585-43cb-955a-ce6750ec1323\") "
Sep 17 16:13:31.898815 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.898792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5eba4e01-c585-43cb-955a-ce6750ec1323" (UID: "5eba4e01-c585-43cb-955a-ce6750ec1323"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:13:31.898865 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.898829    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5eba4e01-c585-43cb-955a-ce6750ec1323" (UID: "5eba4e01-c585-43cb-955a-ce6750ec1323"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:13:31.900490 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.900452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eba4e01-c585-43cb-955a-ce6750ec1323-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5eba4e01-c585-43cb-955a-ce6750ec1323" (UID: "5eba4e01-c585-43cb-955a-ce6750ec1323"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:13:31.900636 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.900619    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-api-access-j9n24" (OuterVolumeSpecName: "kube-api-access-j9n24") pod "5eba4e01-c585-43cb-955a-ce6750ec1323" (UID: "5eba4e01-c585-43cb-955a-ce6750ec1323"). InnerVolumeSpecName "kube-api-access-j9n24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:13:31.909356 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.909319    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eba4e01-c585-43cb-955a-ce6750ec1323-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5eba4e01-c585-43cb-955a-ce6750ec1323" (UID: "5eba4e01-c585-43cb-955a-ce6750ec1323"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:13:31.999668 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.999583    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5eba4e01-c585-43cb-955a-ce6750ec1323-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:13:31.999668 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.999613    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9n24\" (UniqueName: \"kubernetes.io/projected/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-api-access-j9n24\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:13:31.999668 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.999623    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5eba4e01-c585-43cb-955a-ce6750ec1323-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:13:31.999668 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.999632    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:13:31.999668 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:31.999641    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5eba4e01-c585-43cb-955a-ce6750ec1323-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:13:32.704879 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:32.704846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv" event={"ID":"5eba4e01-c585-43cb-955a-ce6750ec1323","Type":"ContainerDied","Data":"08804ad17e136a7368c3e00efb16972a66abf7d0d7b90efa434eebddf4df1152"}
Sep 17 16:13:32.704879 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:32.704873    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"
Sep 17 16:13:32.705455 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:32.704912    2581 scope.go:117] "RemoveContainer" containerID="815a71e7076c707bf6fa43ea2f40f9bc67ed3b60a5b60b6a7514be2bdb8a7249"
Sep 17 16:13:32.717139 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:32.717114    2581 scope.go:117] "RemoveContainer" containerID="00756ea00ee2a31355118619cbcd88dc4b2d6919e3bae0911eb64c85ad7895cf"
Sep 17 16:13:32.725135 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:32.725117    2581 scope.go:117] "RemoveContainer" containerID="974cef0898c1c667ba4aed37a194108177a19befb59d5fac9cc3a0571f734b18"
Sep 17 16:13:32.736313 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:32.736288    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"]
Sep 17 16:13:32.741422 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:32.741396    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-wnwfv"]
Sep 17 16:13:33.557780 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:13:33.557746    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" path="/var/lib/kubelet/pods/5eba4e01-c585-43cb-955a-ce6750ec1323/volumes"
Sep 17 16:15:17.168233 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:17.168197    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"]
Sep 17 16:15:17.168825 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:17.168555    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container" containerID="cri-o://2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a" gracePeriod=30
Sep 17 16:15:17.168825 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:17.168619    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kube-rbac-proxy" containerID="cri-o://d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a" gracePeriod=30
Sep 17 16:15:18.136969 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:18.136939    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerID="d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a" exitCode=2
Sep 17 16:15:18.137134 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:18.137005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" event={"ID":"ff0e821e-7f9c-4b2e-afba-c2fed0704091","Type":"ContainerDied","Data":"d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a"}
Sep 17 16:15:19.275681 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:19.275635    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 17 16:15:21.915804 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:21.915779    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:15:22.044607 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.044577    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff0e821e-7f9c-4b2e-afba-c2fed0704091-proxy-tls\") pod \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") "
Sep 17 16:15:22.044774 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.044642    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-cabundle-cert\") pod \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") "
Sep 17 16:15:22.044774 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.044659    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-rbac-proxy-sar-config\") pod \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") "
Sep 17 16:15:22.044774 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.044684    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fvwr\" (UniqueName: \"kubernetes.io/projected/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-api-access-2fvwr\") pod \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") "
Sep 17 16:15:22.044774 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.044710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kserve-provision-location\") pod \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\" (UID: \"ff0e821e-7f9c-4b2e-afba-c2fed0704091\") "
Sep 17 16:15:22.044973 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.044930    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff0e821e-7f9c-4b2e-afba-c2fed0704091" (UID: "ff0e821e-7f9c-4b2e-afba-c2fed0704091"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:15:22.045042 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.044980    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff0e821e-7f9c-4b2e-afba-c2fed0704091" (UID: "ff0e821e-7f9c-4b2e-afba-c2fed0704091"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:15:22.045176 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.045154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff0e821e-7f9c-4b2e-afba-c2fed0704091" (UID: "ff0e821e-7f9c-4b2e-afba-c2fed0704091"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:15:22.046827 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.046801    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-api-access-2fvwr" (OuterVolumeSpecName: "kube-api-access-2fvwr") pod "ff0e821e-7f9c-4b2e-afba-c2fed0704091" (UID: "ff0e821e-7f9c-4b2e-afba-c2fed0704091"). InnerVolumeSpecName "kube-api-access-2fvwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:15:22.046928 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.046875    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0e821e-7f9c-4b2e-afba-c2fed0704091-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff0e821e-7f9c-4b2e-afba-c2fed0704091" (UID: "ff0e821e-7f9c-4b2e-afba-c2fed0704091"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:15:22.145252 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.145212    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff0e821e-7f9c-4b2e-afba-c2fed0704091-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.145252 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.145243    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.145252 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.145252    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.145252 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.145262    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fvwr\" (UniqueName: \"kubernetes.io/projected/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kube-api-access-2fvwr\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.145551 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.145271    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff0e821e-7f9c-4b2e-afba-c2fed0704091-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:15:22.155138 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.155110    2581 generic.go:356] "Generic (PLEG): container finished" podID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerID="2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a" exitCode=0
Sep 17 16:15:22.155282 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.155174    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"
Sep 17 16:15:22.155282 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.155175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" event={"ID":"ff0e821e-7f9c-4b2e-afba-c2fed0704091","Type":"ContainerDied","Data":"2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a"}
Sep 17 16:15:22.155282 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.155249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k" event={"ID":"ff0e821e-7f9c-4b2e-afba-c2fed0704091","Type":"ContainerDied","Data":"49244796c0264bf9c5ca5e20529ab876a0cda6f71c9d168b7b47000e3ff3dff5"}
Sep 17 16:15:22.155394 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.155288    2581 scope.go:117] "RemoveContainer" containerID="d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a"
Sep 17 16:15:22.164383 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.164355    2581 scope.go:117] "RemoveContainer" containerID="2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a"
Sep 17 16:15:22.172249 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.172231    2581 scope.go:117] "RemoveContainer" containerID="145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389"
Sep 17 16:15:22.178775 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.178752    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"]
Sep 17 16:15:22.180821 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.180804    2581 scope.go:117] "RemoveContainer" containerID="d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a"
Sep 17 16:15:22.181097 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:15:22.181075    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a\": container with ID starting with d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a not found: ID does not exist" containerID="d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a"
Sep 17 16:15:22.181175 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.181103    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a"} err="failed to get container status \"d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a\": rpc error: code = NotFound desc = could not find container \"d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a\": container with ID starting with d749cac928dd03156c2df4a2a12fff4c2d9daab9f9d2cee050776d5fa01bde4a not found: ID does not exist"
Sep 17 16:15:22.181175 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.181123    2581 scope.go:117] "RemoveContainer" containerID="2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a"
Sep 17 16:15:22.181407 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:15:22.181386    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a\": container with ID starting with 2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a not found: ID does not exist" containerID="2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a"
Sep 17 16:15:22.181458 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.181415    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a"} err="failed to get container status \"2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a\": rpc error: code = NotFound desc = could not find container \"2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a\": container with ID starting with 2a25169eacd935ef4841c205701507efc3339e7f347ff4cdb5190b2d2fd69f9a not found: ID does not exist"
Sep 17 16:15:22.181458 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.181431    2581 scope.go:117] "RemoveContainer" containerID="145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389"
Sep 17 16:15:22.181684 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:15:22.181667    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389\": container with ID starting with 145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389 not found: ID does not exist" containerID="145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389"
Sep 17 16:15:22.181727 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.181691    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389"} err="failed to get container status \"145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389\": rpc error: code = NotFound desc = could not find container \"145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389\": container with ID starting with 145a874b7f08df6f6bd7f5b390cd8782d3e7e7a2596dfbc25b6858b43c850389 not found: ID does not exist"
Sep 17 16:15:22.183756 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.183734    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-78f9bdc99b-kvk8k"]
Sep 17 16:15:22.366494 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366401    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"]
Sep 17 16:15:22.366733 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366720    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container"
Sep 17 16:15:22.366778 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366734    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container"
Sep 17 16:15:22.366778 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366749    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kube-rbac-proxy"
Sep 17 16:15:22.366778 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366754    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kube-rbac-proxy"
Sep 17 16:15:22.366778 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366762    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kserve-container"
Sep 17 16:15:22.366778 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366767    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kserve-container"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366781    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="storage-initializer"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366787    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="storage-initializer"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366799    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366804    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366812    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="storage-initializer"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366818    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="storage-initializer"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366863    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kserve-container"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366871    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kube-rbac-proxy"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366878    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eba4e01-c585-43cb-955a-ce6750ec1323" containerName="kserve-container"
Sep 17 16:15:22.366936 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.366885    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" containerName="kube-rbac-proxy"
Sep 17 16:15:22.371329 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.371307    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.373656 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.373635    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 16:15:22.373836 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.373822    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 16:15:22.373917 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.373880    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:15:22.373992 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.373952    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:15:22.374232 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.374215    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:15:22.374285 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.374217    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:15:22.374285 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.374255    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:15:22.386732 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.386708    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"]
Sep 17 16:15:22.447985 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.447950    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbxqb\" (UniqueName: \"kubernetes.io/projected/f6e5976b-8611-462f-8045-e470a0049ff8-kube-api-access-zbxqb\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.448188 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.448004    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6e5976b-8611-462f-8045-e470a0049ff8-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.448188 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.448032    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.448188 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.448102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.448188 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.448180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e5976b-8611-462f-8045-e470a0049ff8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.549450 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.549415    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.549450 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.549455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.549677 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.549530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e5976b-8611-462f-8045-e470a0049ff8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.549677 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.549567    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbxqb\" (UniqueName: \"kubernetes.io/projected/f6e5976b-8611-462f-8045-e470a0049ff8-kube-api-access-zbxqb\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.549677 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.549618    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6e5976b-8611-462f-8045-e470a0049ff8-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.549991 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.549969    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e5976b-8611-462f-8045-e470a0049ff8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.550225 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.550202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-cabundle-cert\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.550357 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.550228    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.552075 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.552060    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6e5976b-8611-462f-8045-e470a0049ff8-proxy-tls\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.562994 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.562974    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbxqb\" (UniqueName: \"kubernetes.io/projected/f6e5976b-8611-462f-8045-e470a0049ff8-kube-api-access-zbxqb\") pod \"isvc-canary-stop-predictor-556d599d48-nhvnp\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.681836 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.681757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:22.952060 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:22.952031    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"]
Sep 17 16:15:22.954562 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:15:22.954535    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e5976b_8611_462f_8045_e470a0049ff8.slice/crio-a11fc65618661cafc0935b692b09571a0c34e4e48ef523e37dbc51a47e2f20ec WatchSource:0}: Error finding container a11fc65618661cafc0935b692b09571a0c34e4e48ef523e37dbc51a47e2f20ec: Status 404 returned error can't find the container with id a11fc65618661cafc0935b692b09571a0c34e4e48ef523e37dbc51a47e2f20ec
Sep 17 16:15:23.162751 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:23.162709    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" event={"ID":"f6e5976b-8611-462f-8045-e470a0049ff8","Type":"ContainerStarted","Data":"30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb"}
Sep 17 16:15:23.162945 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:23.162791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" event={"ID":"f6e5976b-8611-462f-8045-e470a0049ff8","Type":"ContainerStarted","Data":"a11fc65618661cafc0935b692b09571a0c34e4e48ef523e37dbc51a47e2f20ec"}
Sep 17 16:15:23.558056 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:23.558022    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff0e821e-7f9c-4b2e-afba-c2fed0704091" path="/var/lib/kubelet/pods/ff0e821e-7f9c-4b2e-afba-c2fed0704091/volumes"
Sep 17 16:15:23.907016 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:23.906918    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"]
Sep 17 16:15:23.910780 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:23.910758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:23.915297 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:23.915274    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 17 16:15:23.915570 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:23.915555    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 17 16:15:23.922209 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:23.922186    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"]
Sep 17 16:15:24.062370 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.062329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.062370 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.062373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.062796 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.062395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.062796 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.062487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frvv7\" (UniqueName: \"kubernetes.io/projected/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-api-access-frvv7\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.062796 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.062565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.163114 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.163030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.163114 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.163083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.163114 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.163108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.163397 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.163126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.163397 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.163165    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frvv7\" (UniqueName: \"kubernetes.io/projected/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-api-access-frvv7\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.163772 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.163746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.164043 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.164024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.164205 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.164182    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.165849 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.165833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.173228 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.173205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frvv7\" (UniqueName: \"kubernetes.io/projected/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-api-access-frvv7\") pod \"isvc-xgboost-predictor-7566d65bf7-q4ck9\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.221575 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.221535    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:24.495643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:24.495560    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"]
Sep 17 16:15:24.498699 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:15:24.498672    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5e751c_782f_4a1e_ba12_18ea9129a3b0.slice/crio-4581dc352621d01f2d4874a32e78094decc8e1b6f638b8eebe1383e946135127 WatchSource:0}: Error finding container 4581dc352621d01f2d4874a32e78094decc8e1b6f638b8eebe1383e946135127: Status 404 returned error can't find the container with id 4581dc352621d01f2d4874a32e78094decc8e1b6f638b8eebe1383e946135127
Sep 17 16:15:25.173884 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:25.173850    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" event={"ID":"5c5e751c-782f-4a1e-ba12-18ea9129a3b0","Type":"ContainerStarted","Data":"dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb"}
Sep 17 16:15:25.174318 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:25.173935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" event={"ID":"5c5e751c-782f-4a1e-ba12-18ea9129a3b0","Type":"ContainerStarted","Data":"4581dc352621d01f2d4874a32e78094decc8e1b6f638b8eebe1383e946135127"}
Sep 17 16:15:27.183120 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:27.183091    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6e5976b-8611-462f-8045-e470a0049ff8" containerID="30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb" exitCode=0
Sep 17 16:15:27.183433 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:27.183149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" event={"ID":"f6e5976b-8611-462f-8045-e470a0049ff8","Type":"ContainerDied","Data":"30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb"}
Sep 17 16:15:28.189245 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:28.189210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" event={"ID":"f6e5976b-8611-462f-8045-e470a0049ff8","Type":"ContainerStarted","Data":"6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316"}
Sep 17 16:15:28.189734 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:28.189282    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" event={"ID":"f6e5976b-8611-462f-8045-e470a0049ff8","Type":"ContainerStarted","Data":"82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50"}
Sep 17 16:15:28.189734 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:28.189519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:28.212341 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:28.212296    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podStartSLOduration=2.139764656 podStartE2EDuration="6.212281302s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.072516646s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:15:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:15:28.211030449 +0000 UTC m=+2219.134605026" watchObservedRunningTime="2026-09-17 16:15:28.212281302 +0000 UTC m=+2219.135855869"
Sep 17 16:15:29.194358 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:29.194317    2581 generic.go:356] "Generic (PLEG): container finished" podID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerID="dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb" exitCode=0
Sep 17 16:15:29.194787 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:29.194392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" event={"ID":"5c5e751c-782f-4a1e-ba12-18ea9129a3b0","Type":"ContainerDied","Data":"dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb"}
Sep 17 16:15:29.194897 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:29.194837    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:29.196272 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:29.196251    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:15:30.206263 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:30.206186    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:15:35.212051 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:35.212022    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:15:35.212632 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:35.212589    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:15:45.212680 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:45.212633    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:15:50.313521 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:50.313483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" event={"ID":"5c5e751c-782f-4a1e-ba12-18ea9129a3b0","Type":"ContainerStarted","Data":"602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5"}
Sep 17 16:15:50.314003 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:50.313547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" event={"ID":"5c5e751c-782f-4a1e-ba12-18ea9129a3b0","Type":"ContainerStarted","Data":"9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba"}
Sep 17 16:15:50.314003 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:50.313810    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:50.314003 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:50.313929    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:50.315304 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:50.315274    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:15:50.336524 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:50.336484    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podStartSLOduration=2.9438614039999997 podStartE2EDuration="27.336456321s" totalImagesPullingTime="20.545403326s" totalInitContainerRuntime="3.847191591s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:15:50.33498609 +0000 UTC m=+2241.258560659" watchObservedRunningTime="2026-09-17 16:15:50.336456321 +0000 UTC m=+2241.260030887"
Sep 17 16:15:51.318426 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:51.318383    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:15:55.212857 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:55.212760    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:15:56.323861 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:56.323830    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:15:56.324317 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:15:56.324290    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:16:05.212524 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:05.212458    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:16:06.324425 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:06.324383    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:16:15.212728 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:15.212681    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:16:16.325261 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:16.325220    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:16:25.212947 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:25.212905    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:16:26.325011 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:26.324970    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:16:35.212964 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:35.212920    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 17 16:16:36.324275 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:36.324240    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:16:44.939858 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:44.939821    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"]
Sep 17 16:16:44.940255 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:44.940051    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container" containerID="cri-o://82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50" gracePeriod=30
Sep 17 16:16:44.940255 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:44.940152    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kube-rbac-proxy" containerID="cri-o://6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316" gracePeriod=30
Sep 17 16:16:44.940857 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:44.940610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:16:45.206561 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:45.206450    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 17 16:16:45.550753 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:45.550718    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6e5976b-8611-462f-8045-e470a0049ff8" containerID="6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316" exitCode=2
Sep 17 16:16:45.550916 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:45.550772    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" event={"ID":"f6e5976b-8611-462f-8045-e470a0049ff8","Type":"ContainerDied","Data":"6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316"}
Sep 17 16:16:46.325274 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:46.325236    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:16:49.479812 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.479780    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:16:49.506480 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.506431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbxqb\" (UniqueName: \"kubernetes.io/projected/f6e5976b-8611-462f-8045-e470a0049ff8-kube-api-access-zbxqb\") pod \"f6e5976b-8611-462f-8045-e470a0049ff8\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") "
Sep 17 16:16:49.506623 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.506514    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6e5976b-8611-462f-8045-e470a0049ff8-proxy-tls\") pod \"f6e5976b-8611-462f-8045-e470a0049ff8\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") "
Sep 17 16:16:49.506623 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.506561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-cabundle-cert\") pod \"f6e5976b-8611-462f-8045-e470a0049ff8\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") "
Sep 17 16:16:49.506930 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.506864    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-kube-rbac-proxy-sar-config\") pod \"f6e5976b-8611-462f-8045-e470a0049ff8\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") "
Sep 17 16:16:49.506930 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.506884    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f6e5976b-8611-462f-8045-e470a0049ff8" (UID: "f6e5976b-8611-462f-8045-e470a0049ff8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:16:49.506930 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.506928    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e5976b-8611-462f-8045-e470a0049ff8-kserve-provision-location\") pod \"f6e5976b-8611-462f-8045-e470a0049ff8\" (UID: \"f6e5976b-8611-462f-8045-e470a0049ff8\") "
Sep 17 16:16:49.507276 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.507171    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f6e5976b-8611-462f-8045-e470a0049ff8" (UID: "f6e5976b-8611-462f-8045-e470a0049ff8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:16:49.507276 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.507211    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6e5976b-8611-462f-8045-e470a0049ff8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f6e5976b-8611-462f-8045-e470a0049ff8" (UID: "f6e5976b-8611-462f-8045-e470a0049ff8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:16:49.508955 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.508924    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e5976b-8611-462f-8045-e470a0049ff8-kube-api-access-zbxqb" (OuterVolumeSpecName: "kube-api-access-zbxqb") pod "f6e5976b-8611-462f-8045-e470a0049ff8" (UID: "f6e5976b-8611-462f-8045-e470a0049ff8"). InnerVolumeSpecName "kube-api-access-zbxqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:16:49.509059 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.508991    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6e5976b-8611-462f-8045-e470a0049ff8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f6e5976b-8611-462f-8045-e470a0049ff8" (UID: "f6e5976b-8611-462f-8045-e470a0049ff8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:16:49.572590 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.572527    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6e5976b-8611-462f-8045-e470a0049ff8" containerID="82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50" exitCode=0
Sep 17 16:16:49.572590 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.572577    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"
Sep 17 16:16:49.572760 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.572611    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" event={"ID":"f6e5976b-8611-462f-8045-e470a0049ff8","Type":"ContainerDied","Data":"82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50"}
Sep 17 16:16:49.572760 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.572679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp" event={"ID":"f6e5976b-8611-462f-8045-e470a0049ff8","Type":"ContainerDied","Data":"a11fc65618661cafc0935b692b09571a0c34e4e48ef523e37dbc51a47e2f20ec"}
Sep 17 16:16:49.572760 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.572717    2581 scope.go:117] "RemoveContainer" containerID="6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316"
Sep 17 16:16:49.580693 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.580678    2581 scope.go:117] "RemoveContainer" containerID="82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50"
Sep 17 16:16:49.587937 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.587918    2581 scope.go:117] "RemoveContainer" containerID="30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb"
Sep 17 16:16:49.594977 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.594961    2581 scope.go:117] "RemoveContainer" containerID="6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316"
Sep 17 16:16:49.595203 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:16:49.595186    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316\": container with ID starting with 6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316 not found: ID does not exist" containerID="6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316"
Sep 17 16:16:49.595253 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.595217    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316"} err="failed to get container status \"6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316\": rpc error: code = NotFound desc = could not find container \"6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316\": container with ID starting with 6966f54bd354076fd25d3129ba3b9dfb10629ddec4256c6d41ed8d7bb06f0316 not found: ID does not exist"
Sep 17 16:16:49.595253 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.595240    2581 scope.go:117] "RemoveContainer" containerID="82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50"
Sep 17 16:16:49.595506 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:16:49.595458    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50\": container with ID starting with 82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50 not found: ID does not exist" containerID="82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50"
Sep 17 16:16:49.595557 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.595498    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50"} err="failed to get container status \"82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50\": rpc error: code = NotFound desc = could not find container \"82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50\": container with ID starting with 82b8e81a6098c7a5bcb6cc1b6bb84596be397daabb2019b5dbac8a155ef70c50 not found: ID does not exist"
Sep 17 16:16:49.595557 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.595518    2581 scope.go:117] "RemoveContainer" containerID="30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb"
Sep 17 16:16:49.595768 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:16:49.595750    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb\": container with ID starting with 30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb not found: ID does not exist" containerID="30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb"
Sep 17 16:16:49.595832 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.595778    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb"} err="failed to get container status \"30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb\": rpc error: code = NotFound desc = could not find container \"30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb\": container with ID starting with 30521ed2c8a6ad553dd43b66618fdd00ed0e8dab4a3139f12ca4e4de665b6feb not found: ID does not exist"
Sep 17 16:16:49.601600 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.601579    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"]
Sep 17 16:16:49.605717 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.605699    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-556d599d48-nhvnp"]
Sep 17 16:16:49.608552 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.608536    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f6e5976b-8611-462f-8045-e470a0049ff8-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:16:49.608613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.608554    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:16:49.608613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.608565    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f6e5976b-8611-462f-8045-e470a0049ff8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:16:49.608613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.608575    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f6e5976b-8611-462f-8045-e470a0049ff8-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:16:49.608613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:49.608583    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbxqb\" (UniqueName: \"kubernetes.io/projected/f6e5976b-8611-462f-8045-e470a0049ff8-kube-api-access-zbxqb\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:16:51.557598 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:51.557569    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" path="/var/lib/kubelet/pods/f6e5976b-8611-462f-8045-e470a0049ff8/volumes"
Sep 17 16:16:56.325657 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:16:56.325628    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:17:04.478555 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:04.478523    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"]
Sep 17 16:17:04.478987 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:04.478778    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kube-rbac-proxy" containerID="cri-o://602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5" gracePeriod=30
Sep 17 16:17:04.478987 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:04.478767    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" containerID="cri-o://9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba" gracePeriod=30
Sep 17 16:17:04.638418 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:04.638384    2581 generic.go:356] "Generic (PLEG): container finished" podID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerID="602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5" exitCode=2
Sep 17 16:17:04.638585 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:04.638449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" event={"ID":"5c5e751c-782f-4a1e-ba12-18ea9129a3b0","Type":"ContainerDied","Data":"602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5"}
Sep 17 16:17:06.319027 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:06.318980    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 17 16:17:06.324264 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:06.324237    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 17 16:17:08.621373 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.621349    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:17:08.657058 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.657025    2581 generic.go:356] "Generic (PLEG): container finished" podID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerID="9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba" exitCode=0
Sep 17 16:17:08.657219 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.657084    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"
Sep 17 16:17:08.657219 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.657101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" event={"ID":"5c5e751c-782f-4a1e-ba12-18ea9129a3b0","Type":"ContainerDied","Data":"9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba"}
Sep 17 16:17:08.657219 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.657190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9" event={"ID":"5c5e751c-782f-4a1e-ba12-18ea9129a3b0","Type":"ContainerDied","Data":"4581dc352621d01f2d4874a32e78094decc8e1b6f638b8eebe1383e946135127"}
Sep 17 16:17:08.657219 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.657219    2581 scope.go:117] "RemoveContainer" containerID="602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5"
Sep 17 16:17:08.665281 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.665264    2581 scope.go:117] "RemoveContainer" containerID="9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba"
Sep 17 16:17:08.672321 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.672301    2581 scope.go:117] "RemoveContainer" containerID="dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb"
Sep 17 16:17:08.679311 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.679290    2581 scope.go:117] "RemoveContainer" containerID="602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5"
Sep 17 16:17:08.679588 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:17:08.679571    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5\": container with ID starting with 602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5 not found: ID does not exist" containerID="602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5"
Sep 17 16:17:08.679651 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.679600    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5"} err="failed to get container status \"602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5\": rpc error: code = NotFound desc = could not find container \"602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5\": container with ID starting with 602ede95f08294ddc363ea17591d03d5625c09470b991b8d97750d5d39835ee5 not found: ID does not exist"
Sep 17 16:17:08.679651 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.679618    2581 scope.go:117] "RemoveContainer" containerID="9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba"
Sep 17 16:17:08.679868 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:17:08.679849    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba\": container with ID starting with 9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba not found: ID does not exist" containerID="9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba"
Sep 17 16:17:08.679912 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.679875    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba"} err="failed to get container status \"9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba\": rpc error: code = NotFound desc = could not find container \"9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba\": container with ID starting with 9e50f54b205b18d98cc814f7b10098149ef63ebf0f9e9d60ea0d73eb285343ba not found: ID does not exist"
Sep 17 16:17:08.679912 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.679891    2581 scope.go:117] "RemoveContainer" containerID="dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb"
Sep 17 16:17:08.680117 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:17:08.680100    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb\": container with ID starting with dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb not found: ID does not exist" containerID="dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb"
Sep 17 16:17:08.680159 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.680121    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb"} err="failed to get container status \"dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb\": rpc error: code = NotFound desc = could not find container \"dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb\": container with ID starting with dc607df3184c6b27ac5ab7e17d9efd3cd012b5add7cc3896f935bd1c1063b8fb not found: ID does not exist"
Sep 17 16:17:08.771298 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.771261    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frvv7\" (UniqueName: \"kubernetes.io/projected/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-api-access-frvv7\") pod \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") "
Sep 17 16:17:08.771515 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.771317    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kserve-provision-location\") pod \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") "
Sep 17 16:17:08.771515 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.771348    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-rbac-proxy-sar-config\") pod \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") "
Sep 17 16:17:08.771515 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.771446    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-proxy-tls\") pod \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") "
Sep 17 16:17:08.771694 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.771562    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-cabundle-cert\") pod \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\" (UID: \"5c5e751c-782f-4a1e-ba12-18ea9129a3b0\") "
Sep 17 16:17:08.771759 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.771724    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c5e751c-782f-4a1e-ba12-18ea9129a3b0" (UID: "5c5e751c-782f-4a1e-ba12-18ea9129a3b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:17:08.771759 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.771742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c5e751c-782f-4a1e-ba12-18ea9129a3b0" (UID: "5c5e751c-782f-4a1e-ba12-18ea9129a3b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:08.771866 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.771838    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:17:08.771866 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.771857    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:17:08.772077 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.772038    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c5e751c-782f-4a1e-ba12-18ea9129a3b0" (UID: "5c5e751c-782f-4a1e-ba12-18ea9129a3b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:17:08.773618 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.773587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c5e751c-782f-4a1e-ba12-18ea9129a3b0" (UID: "5c5e751c-782f-4a1e-ba12-18ea9129a3b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:17:08.773720 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.773621    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-api-access-frvv7" (OuterVolumeSpecName: "kube-api-access-frvv7") pod "5c5e751c-782f-4a1e-ba12-18ea9129a3b0" (UID: "5c5e751c-782f-4a1e-ba12-18ea9129a3b0"). InnerVolumeSpecName "kube-api-access-frvv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:17:08.873205 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.873152    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frvv7\" (UniqueName: \"kubernetes.io/projected/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-kube-api-access-frvv7\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:17:08.873205 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.873199    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:17:08.873205 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.873209    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c5e751c-782f-4a1e-ba12-18ea9129a3b0-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:17:08.980002 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.979970    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"]
Sep 17 16:17:08.984547 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:08.984522    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-q4ck9"]
Sep 17 16:17:09.557144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.557113    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" path="/var/lib/kubelet/pods/5c5e751c-782f-4a1e-ba12-18ea9129a3b0/volumes"
Sep 17 16:17:09.687157 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687121    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"]
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687427    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="storage-initializer"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687437    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="storage-initializer"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687444    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="storage-initializer"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687450    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="storage-initializer"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687477    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kube-rbac-proxy"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687483    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kube-rbac-proxy"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687494    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687500    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687510    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687515    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687525    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kube-rbac-proxy"
Sep 17 16:17:09.687554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687530    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kube-rbac-proxy"
Sep 17 16:17:09.687970 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687576    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kserve-container"
Sep 17 16:17:09.687970 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687585    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kube-rbac-proxy"
Sep 17 16:17:09.687970 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687593    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c5e751c-782f-4a1e-ba12-18ea9129a3b0" containerName="kube-rbac-proxy"
Sep 17 16:17:09.687970 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.687599    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e5976b-8611-462f-8045-e470a0049ff8" containerName="kserve-container"
Sep 17 16:17:09.692056 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.692038    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.694065 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.694043    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:17:09.694202 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.694162    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 16:17:09.694392 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.694363    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:17:09.694392 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.694384    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 16:17:09.694575 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.694391    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:17:09.694575 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.694450    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:17:09.694722 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.694705    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:17:09.703143 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.703122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"]
Sep 17 16:17:09.781238 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.781204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.781421 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.781267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.781421 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.781312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.781421 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.781334    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.781421 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.781412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvptz\" (UniqueName: \"kubernetes.io/projected/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-api-access-qvptz\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.882739 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.882641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvptz\" (UniqueName: \"kubernetes.io/projected/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-api-access-qvptz\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.882739 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.882704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.882739 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.882740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.883037 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.882762    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.883037 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.882779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.883215 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.883196    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.883514 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.883478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.883571 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.883533    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.885366 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.885347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:09.891060 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:09.891040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvptz\" (UniqueName: \"kubernetes.io/projected/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-api-access-qvptz\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:10.002513 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:10.002436    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:10.281346 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:10.281194    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"]
Sep 17 16:17:10.283935 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:17:10.283905    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d04bcc3_d2a2_4e86_aabb_43f21c406dee.slice/crio-c4260d447748ced024a8084080e3c929774b505ce01ab060089a5f0cdcbd1c36 WatchSource:0}: Error finding container c4260d447748ced024a8084080e3c929774b505ce01ab060089a5f0cdcbd1c36: Status 404 returned error can't find the container with id c4260d447748ced024a8084080e3c929774b505ce01ab060089a5f0cdcbd1c36
Sep 17 16:17:10.668177 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:10.668138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" event={"ID":"4d04bcc3-d2a2-4e86-aabb-43f21c406dee","Type":"ContainerStarted","Data":"770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd"}
Sep 17 16:17:10.668362 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:10.668198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" event={"ID":"4d04bcc3-d2a2-4e86-aabb-43f21c406dee","Type":"ContainerStarted","Data":"c4260d447748ced024a8084080e3c929774b505ce01ab060089a5f0cdcbd1c36"}
Sep 17 16:17:14.684909 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:14.684874    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerID="770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd" exitCode=0
Sep 17 16:17:14.685300 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:14.684948    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" event={"ID":"4d04bcc3-d2a2-4e86-aabb-43f21c406dee","Type":"ContainerDied","Data":"770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd"}
Sep 17 16:17:15.691040 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:15.691010    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" event={"ID":"4d04bcc3-d2a2-4e86-aabb-43f21c406dee","Type":"ContainerStarted","Data":"2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a"}
Sep 17 16:17:15.691430 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:15.691068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" event={"ID":"4d04bcc3-d2a2-4e86-aabb-43f21c406dee","Type":"ContainerStarted","Data":"e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7"}
Sep 17 16:17:15.691430 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:15.691254    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:15.714419 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:15.714355    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" podStartSLOduration=2.798560445 podStartE2EDuration="6.714339177s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.915778732s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:17:15.712690178 +0000 UTC m=+2326.636264746" watchObservedRunningTime="2026-09-17 16:17:15.714339177 +0000 UTC m=+2326.637913745"
Sep 17 16:17:16.694947 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:16.694915    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:22.704276 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:22.704195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:52.708239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:52.708209    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:17:59.811260 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:59.811225    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"]
Sep 17 16:17:59.811782 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:59.811453    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kserve-container" containerID="cri-o://e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7" gracePeriod=30
Sep 17 16:17:59.811782 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:17:59.811512    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kube-rbac-proxy" containerID="cri-o://2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a" gracePeriod=30
Sep 17 16:18:00.886853 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:00.886818    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerID="2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a" exitCode=2
Sep 17 16:18:00.887220 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:00.886879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" event={"ID":"4d04bcc3-d2a2-4e86-aabb-43f21c406dee","Type":"ContainerDied","Data":"2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a"}
Sep 17 16:18:02.700005 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:02.699961    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 17 16:18:02.705283 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:02.705250    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.47:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 17 16:18:04.939113 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:04.939072    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"]
Sep 17 16:18:04.942671 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:04.942656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:04.945372 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:04.945350    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 17 16:18:04.945515 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:04.945356    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 17 16:18:04.955925 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:04.955903    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"]
Sep 17 16:18:05.016892 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.016855    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.017040 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.016901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd72918a-324f-4736-8d02-b06a6d6b3483-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.017040 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.016961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rgg\" (UniqueName: \"kubernetes.io/projected/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-api-access-h8rgg\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.017040 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.016988    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.017040 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.017007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd72918a-324f-4736-8d02-b06a6d6b3483-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.117947 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.117907    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.118131 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.117958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd72918a-324f-4736-8d02-b06a6d6b3483-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.118131 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.118000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rgg\" (UniqueName: \"kubernetes.io/projected/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-api-access-h8rgg\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.118131 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.118029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.118131 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.118068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd72918a-324f-4736-8d02-b06a6d6b3483-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.118359 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:18:05.118169    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-serving-cert: secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 16:18:05.118359 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:18:05.118238    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd72918a-324f-4736-8d02-b06a6d6b3483-proxy-tls podName:bd72918a-324f-4736-8d02-b06a6d6b3483 nodeName:}" failed. No retries permitted until 2026-09-17 16:18:05.61822003 +0000 UTC m=+2376.541794595 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd72918a-324f-4736-8d02-b06a6d6b3483-proxy-tls") pod "xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" (UID: "bd72918a-324f-4736-8d02-b06a6d6b3483") : secret "xgboost-v2-mlserver-predictor-serving-cert" not found
Sep 17 16:18:05.118513 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.118488    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd72918a-324f-4736-8d02-b06a6d6b3483-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.118800 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.118776    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.118874 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.118795    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.127596 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.127570    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rgg\" (UniqueName: \"kubernetes.io/projected/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-api-access-h8rgg\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.622568 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.622526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd72918a-324f-4736-8d02-b06a6d6b3483-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.624986 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.624963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd72918a-324f-4736-8d02-b06a6d6b3483-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:05.852168 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:05.852129    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:06.125990 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.125962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"]
Sep 17 16:18:06.128414 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:18:06.128384    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd72918a_324f_4736_8d02_b06a6d6b3483.slice/crio-9292b413bb8c019ee43ae85d470d3afa83bd35d4d1e8d801d48ea22441b69774 WatchSource:0}: Error finding container 9292b413bb8c019ee43ae85d470d3afa83bd35d4d1e8d801d48ea22441b69774: Status 404 returned error can't find the container with id 9292b413bb8c019ee43ae85d470d3afa83bd35d4d1e8d801d48ea22441b69774
Sep 17 16:18:06.450458 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.450437    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:18:06.529490 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.529450    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-proxy-tls\") pod \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") "
Sep 17 16:18:06.529696 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.529504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kserve-provision-location\") pod \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") "
Sep 17 16:18:06.529696 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.529533    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-cabundle-cert\") pod \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") "
Sep 17 16:18:06.529696 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.529559    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvptz\" (UniqueName: \"kubernetes.io/projected/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-api-access-qvptz\") pod \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") "
Sep 17 16:18:06.529696 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.529627    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-rbac-proxy-sar-config\") pod \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\" (UID: \"4d04bcc3-d2a2-4e86-aabb-43f21c406dee\") "
Sep 17 16:18:06.529925 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.529859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4d04bcc3-d2a2-4e86-aabb-43f21c406dee" (UID: "4d04bcc3-d2a2-4e86-aabb-43f21c406dee"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:18:06.529925 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.529882    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4d04bcc3-d2a2-4e86-aabb-43f21c406dee" (UID: "4d04bcc3-d2a2-4e86-aabb-43f21c406dee"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:06.530091 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.530071    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4d04bcc3-d2a2-4e86-aabb-43f21c406dee" (UID: "4d04bcc3-d2a2-4e86-aabb-43f21c406dee"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:18:06.531676 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.531656    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-api-access-qvptz" (OuterVolumeSpecName: "kube-api-access-qvptz") pod "4d04bcc3-d2a2-4e86-aabb-43f21c406dee" (UID: "4d04bcc3-d2a2-4e86-aabb-43f21c406dee"). InnerVolumeSpecName "kube-api-access-qvptz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:18:06.532041 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.532022    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d04bcc3-d2a2-4e86-aabb-43f21c406dee" (UID: "4d04bcc3-d2a2-4e86-aabb-43f21c406dee"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:18:06.630764 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.630733    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:18:06.630764 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.630760    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:18:06.630764 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.630771    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:18:06.630976 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.630781    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:18:06.630976 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.630790    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvptz\" (UniqueName: \"kubernetes.io/projected/4d04bcc3-d2a2-4e86-aabb-43f21c406dee-kube-api-access-qvptz\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:18:06.914899 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.914866    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerID="e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7" exitCode=0
Sep 17 16:18:06.915093 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.914926    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"
Sep 17 16:18:06.915093 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.914952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" event={"ID":"4d04bcc3-d2a2-4e86-aabb-43f21c406dee","Type":"ContainerDied","Data":"e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7"}
Sep 17 16:18:06.915093 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.915018    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9" event={"ID":"4d04bcc3-d2a2-4e86-aabb-43f21c406dee","Type":"ContainerDied","Data":"c4260d447748ced024a8084080e3c929774b505ce01ab060089a5f0cdcbd1c36"}
Sep 17 16:18:06.915093 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.915036    2581 scope.go:117] "RemoveContainer" containerID="2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a"
Sep 17 16:18:06.916596 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.916576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" event={"ID":"bd72918a-324f-4736-8d02-b06a6d6b3483","Type":"ContainerStarted","Data":"6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156"}
Sep 17 16:18:06.916682 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.916621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" event={"ID":"bd72918a-324f-4736-8d02-b06a6d6b3483","Type":"ContainerStarted","Data":"9292b413bb8c019ee43ae85d470d3afa83bd35d4d1e8d801d48ea22441b69774"}
Sep 17 16:18:06.924169 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.924151    2581 scope.go:117] "RemoveContainer" containerID="e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7"
Sep 17 16:18:06.933624 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.933598    2581 scope.go:117] "RemoveContainer" containerID="770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd"
Sep 17 16:18:06.940995 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.940973    2581 scope.go:117] "RemoveContainer" containerID="2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a"
Sep 17 16:18:06.941372 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:18:06.941350    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a\": container with ID starting with 2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a not found: ID does not exist" containerID="2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a"
Sep 17 16:18:06.941496 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.941379    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a"} err="failed to get container status \"2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a\": rpc error: code = NotFound desc = could not find container \"2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a\": container with ID starting with 2e9822cd3e0ecd5be4224d400f1f7f3bda48da17694ecb7f4fd27f037376760a not found: ID does not exist"
Sep 17 16:18:06.941496 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.941396    2581 scope.go:117] "RemoveContainer" containerID="e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7"
Sep 17 16:18:06.941667 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:18:06.941651    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7\": container with ID starting with e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7 not found: ID does not exist" containerID="e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7"
Sep 17 16:18:06.941729 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.941671    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7"} err="failed to get container status \"e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7\": rpc error: code = NotFound desc = could not find container \"e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7\": container with ID starting with e46d81a02fcf3e17d677c71894767230fa579e1fcaa02a65744714d3b66e28d7 not found: ID does not exist"
Sep 17 16:18:06.941729 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.941685    2581 scope.go:117] "RemoveContainer" containerID="770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd"
Sep 17 16:18:06.941914 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:18:06.941899    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd\": container with ID starting with 770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd not found: ID does not exist" containerID="770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd"
Sep 17 16:18:06.941988 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.941916    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd"} err="failed to get container status \"770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd\": rpc error: code = NotFound desc = could not find container \"770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd\": container with ID starting with 770ff265ff616466b3fcd4a54d26114e03ce92f6e8185a23d66bd0ee77ab15bd not found: ID does not exist"
Sep 17 16:18:06.955966 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.955946    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"]
Sep 17 16:18:06.960504 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:06.960484    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-ct5k9"]
Sep 17 16:18:07.557785 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:07.557752    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" path="/var/lib/kubelet/pods/4d04bcc3-d2a2-4e86-aabb-43f21c406dee/volumes"
Sep 17 16:18:10.935664 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:10.935633    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerID="6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156" exitCode=0
Sep 17 16:18:10.936037 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:10.935706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" event={"ID":"bd72918a-324f-4736-8d02-b06a6d6b3483","Type":"ContainerDied","Data":"6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156"}
Sep 17 16:18:11.941811 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:11.941776    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" event={"ID":"bd72918a-324f-4736-8d02-b06a6d6b3483","Type":"ContainerStarted","Data":"36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b"}
Sep 17 16:18:11.942293 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:11.941849    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" event={"ID":"bd72918a-324f-4736-8d02-b06a6d6b3483","Type":"ContainerStarted","Data":"9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0"}
Sep 17 16:18:11.942293 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:11.942071    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:11.965359 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:11.965311    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" podStartSLOduration=3.971469071 podStartE2EDuration="7.96529545s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.993826379s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:18:11.963436182 +0000 UTC m=+2382.887010752" watchObservedRunningTime="2026-09-17 16:18:11.96529545 +0000 UTC m=+2382.888870016"
Sep 17 16:18:12.946441 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:12.946405    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:18.955849 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:18.955818    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:48.959826 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:48.959788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:18:55.009427 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:55.009394    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"]
Sep 17 16:18:55.009832 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:55.009673    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kserve-container" containerID="cri-o://9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0" gracePeriod=30
Sep 17 16:18:55.009832 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:55.009706    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kube-rbac-proxy" containerID="cri-o://36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b" gracePeriod=30
Sep 17 16:18:55.126762 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:55.126689    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerID="36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b" exitCode=2
Sep 17 16:18:55.126901 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:55.126758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" event={"ID":"bd72918a-324f-4736-8d02-b06a6d6b3483","Type":"ContainerDied","Data":"36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b"}
Sep 17 16:18:58.951125 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:58.951083    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 17 16:18:58.956599 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:18:58.956568    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.48:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 17 16:19:01.648506 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.648459    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:19:01.696195 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd72918a-324f-4736-8d02-b06a6d6b3483-proxy-tls\") pod \"bd72918a-324f-4736-8d02-b06a6d6b3483\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") "
Sep 17 16:19:01.696195 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696165    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-rbac-proxy-sar-config\") pod \"bd72918a-324f-4736-8d02-b06a6d6b3483\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") "
Sep 17 16:19:01.696195 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696193    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd72918a-324f-4736-8d02-b06a6d6b3483-kserve-provision-location\") pod \"bd72918a-324f-4736-8d02-b06a6d6b3483\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") "
Sep 17 16:19:01.696486 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696346    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8rgg\" (UniqueName: \"kubernetes.io/projected/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-api-access-h8rgg\") pod \"bd72918a-324f-4736-8d02-b06a6d6b3483\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") "
Sep 17 16:19:01.696486 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696441    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-cabundle-cert\") pod \"bd72918a-324f-4736-8d02-b06a6d6b3483\" (UID: \"bd72918a-324f-4736-8d02-b06a6d6b3483\") "
Sep 17 16:19:01.696599 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696510    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd72918a-324f-4736-8d02-b06a6d6b3483-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd72918a-324f-4736-8d02-b06a6d6b3483" (UID: "bd72918a-324f-4736-8d02-b06a6d6b3483"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:19:01.696599 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696546    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd72918a-324f-4736-8d02-b06a6d6b3483" (UID: "bd72918a-324f-4736-8d02-b06a6d6b3483"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:19:01.696734 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696713    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:19:01.696734 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696736    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd72918a-324f-4736-8d02-b06a6d6b3483-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:19:01.696913 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.696784    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd72918a-324f-4736-8d02-b06a6d6b3483" (UID: "bd72918a-324f-4736-8d02-b06a6d6b3483"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:19:01.698329 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.698308    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd72918a-324f-4736-8d02-b06a6d6b3483-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd72918a-324f-4736-8d02-b06a6d6b3483" (UID: "bd72918a-324f-4736-8d02-b06a6d6b3483"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:19:01.698449 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.698339    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-api-access-h8rgg" (OuterVolumeSpecName: "kube-api-access-h8rgg") pod "bd72918a-324f-4736-8d02-b06a6d6b3483" (UID: "bd72918a-324f-4736-8d02-b06a6d6b3483"). InnerVolumeSpecName "kube-api-access-h8rgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:19:01.798096 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.798063    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd72918a-324f-4736-8d02-b06a6d6b3483-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:19:01.798096 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.798090    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rgg\" (UniqueName: \"kubernetes.io/projected/bd72918a-324f-4736-8d02-b06a6d6b3483-kube-api-access-h8rgg\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:19:01.798096 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:01.798099    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd72918a-324f-4736-8d02-b06a6d6b3483-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:19:02.159480 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.159434    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerID="9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0" exitCode=0
Sep 17 16:19:02.159633 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.159509    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"
Sep 17 16:19:02.159633 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.159523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" event={"ID":"bd72918a-324f-4736-8d02-b06a6d6b3483","Type":"ContainerDied","Data":"9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0"}
Sep 17 16:19:02.159633 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.159583    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw" event={"ID":"bd72918a-324f-4736-8d02-b06a6d6b3483","Type":"ContainerDied","Data":"9292b413bb8c019ee43ae85d470d3afa83bd35d4d1e8d801d48ea22441b69774"}
Sep 17 16:19:02.159633 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.159609    2581 scope.go:117] "RemoveContainer" containerID="36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b"
Sep 17 16:19:02.168628 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.168610    2581 scope.go:117] "RemoveContainer" containerID="9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0"
Sep 17 16:19:02.176341 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.176325    2581 scope.go:117] "RemoveContainer" containerID="6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156"
Sep 17 16:19:02.182715 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.182687    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"]
Sep 17 16:19:02.184613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.184584    2581 scope.go:117] "RemoveContainer" containerID="36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b"
Sep 17 16:19:02.184892 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:19:02.184873    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b\": container with ID starting with 36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b not found: ID does not exist" containerID="36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b"
Sep 17 16:19:02.184938 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.184903    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b"} err="failed to get container status \"36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b\": rpc error: code = NotFound desc = could not find container \"36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b\": container with ID starting with 36d1d47f792b773428d6809aee887cc83177433684b07cad299a13f6a835ea9b not found: ID does not exist"
Sep 17 16:19:02.184938 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.184923    2581 scope.go:117] "RemoveContainer" containerID="9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0"
Sep 17 16:19:02.185191 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:19:02.185171    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0\": container with ID starting with 9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0 not found: ID does not exist" containerID="9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0"
Sep 17 16:19:02.185239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.185197    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0"} err="failed to get container status \"9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0\": rpc error: code = NotFound desc = could not find container \"9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0\": container with ID starting with 9e72cf1f6829a2dff803d5e44f3abd478e7720d6b275af81345fa59e5ea276c0 not found: ID does not exist"
Sep 17 16:19:02.185239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.185212    2581 scope.go:117] "RemoveContainer" containerID="6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156"
Sep 17 16:19:02.185435 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:19:02.185419    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156\": container with ID starting with 6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156 not found: ID does not exist" containerID="6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156"
Sep 17 16:19:02.185512 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.185442    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156"} err="failed to get container status \"6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156\": rpc error: code = NotFound desc = could not find container \"6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156\": container with ID starting with 6bbf1b4fe4f07948f375b64cad4f86fa1b0e198c232fa803a7b2012142833156 not found: ID does not exist"
Sep 17 16:19:02.191446 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:02.191423    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-dpdrw"]
Sep 17 16:19:03.557837 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:03.557807    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" path="/var/lib/kubelet/pods/bd72918a-324f-4736-8d02-b06a6d6b3483/volumes"
Sep 17 16:19:08.798481 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798430    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"]
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798745    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kube-rbac-proxy"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798756    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kube-rbac-proxy"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798770    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kserve-container"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798776    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kserve-container"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798788    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="storage-initializer"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798794    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="storage-initializer"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798802    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="storage-initializer"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798807    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="storage-initializer"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798812    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kube-rbac-proxy"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798817    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kube-rbac-proxy"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798830    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kserve-container"
Sep 17 16:19:08.798844 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798835    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kserve-container"
Sep 17 16:19:08.799222 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798885    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kube-rbac-proxy"
Sep 17 16:19:08.799222 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798893    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kube-rbac-proxy"
Sep 17 16:19:08.799222 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798899    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d04bcc3-d2a2-4e86-aabb-43f21c406dee" containerName="kserve-container"
Sep 17 16:19:08.799222 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.798905    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd72918a-324f-4736-8d02-b06a6d6b3483" containerName="kserve-container"
Sep 17 16:19:08.801890 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.801872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.805954 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.805920    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:19:08.805954 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.805925    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:19:08.806144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.805926    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:19:08.806144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.805926    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:19:08.806144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.805928    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 17 16:19:08.807995 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.807977    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:19:08.808520 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.808213    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:19:08.818357 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.818323    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"]
Sep 17 16:19:08.857430 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.857391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7fdc784-4469-4210-b8af-7ed605a20047-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.857620 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.857447    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5q5v\" (UniqueName: \"kubernetes.io/projected/e7fdc784-4469-4210-b8af-7ed605a20047-kube-api-access-x5q5v\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.857620 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.857506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7fdc784-4469-4210-b8af-7ed605a20047-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.857620 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.857547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.857754 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.857625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.958957 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.958921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5q5v\" (UniqueName: \"kubernetes.io/projected/e7fdc784-4469-4210-b8af-7ed605a20047-kube-api-access-x5q5v\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.958957 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.958960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7fdc784-4469-4210-b8af-7ed605a20047-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.959186 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.958994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.959186 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.959027    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.959186 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.959043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7fdc784-4469-4210-b8af-7ed605a20047-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.959635 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.959608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7fdc784-4469-4210-b8af-7ed605a20047-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.959765 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.959747    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.959851 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.959833    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.961402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.961386    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7fdc784-4469-4210-b8af-7ed605a20047-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:08.968196 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:08.968177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5q5v\" (UniqueName: \"kubernetes.io/projected/e7fdc784-4469-4210-b8af-7ed605a20047-kube-api-access-x5q5v\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:09.112124 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:09.112030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:09.389859 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:09.389783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"]
Sep 17 16:19:09.392753 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:19:09.392729    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7fdc784_4469_4210_b8af_7ed605a20047.slice/crio-b010640665abfc050e0a6e5f47b2b9a6ead9d793ae2ceb3613b347014a501bfd WatchSource:0}: Error finding container b010640665abfc050e0a6e5f47b2b9a6ead9d793ae2ceb3613b347014a501bfd: Status 404 returned error can't find the container with id b010640665abfc050e0a6e5f47b2b9a6ead9d793ae2ceb3613b347014a501bfd
Sep 17 16:19:10.195681 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:10.195640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" event={"ID":"e7fdc784-4469-4210-b8af-7ed605a20047","Type":"ContainerStarted","Data":"69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157"}
Sep 17 16:19:10.196064 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:10.195713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" event={"ID":"e7fdc784-4469-4210-b8af-7ed605a20047","Type":"ContainerStarted","Data":"b010640665abfc050e0a6e5f47b2b9a6ead9d793ae2ceb3613b347014a501bfd"}
Sep 17 16:19:14.214496 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:14.214439    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7fdc784-4469-4210-b8af-7ed605a20047" containerID="69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157" exitCode=0
Sep 17 16:19:14.214876 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:14.214519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" event={"ID":"e7fdc784-4469-4210-b8af-7ed605a20047","Type":"ContainerDied","Data":"69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157"}
Sep 17 16:19:14.215746 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:14.215732    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:19:21.250518 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:21.250480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" event={"ID":"e7fdc784-4469-4210-b8af-7ed605a20047","Type":"ContainerStarted","Data":"448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0"}
Sep 17 16:19:21.250938 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:21.250543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" event={"ID":"e7fdc784-4469-4210-b8af-7ed605a20047","Type":"ContainerStarted","Data":"39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e"}
Sep 17 16:19:21.250938 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:21.250721    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:21.271200 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:21.271147    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podStartSLOduration=3.354629958 podStartE2EDuration="13.271133116s" totalImagesPullingTime="6.14241544s" totalInitContainerRuntime="3.774087718s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:19:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:19:21.269755059 +0000 UTC m=+2452.193329626" watchObservedRunningTime="2026-09-17 16:19:21.271133116 +0000 UTC m=+2452.194707682"
Sep 17 16:19:22.255706 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:22.255674    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:22.256661 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:22.256636    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:19:23.259496 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:23.259444    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:19:28.264055 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:28.264026    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:19:28.264549 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:28.264518    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:19:38.265397 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:38.265358    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:19:48.265011 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:48.264971    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:19:58.265225 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:19:58.265188    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:20:08.265167 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:08.265126    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:20:18.265392 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:18.265349    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:20:28.264943 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:28.264903    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:20:38.264484 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:38.264432    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:20:40.553586 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:40.553542    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:20:50.554415 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:50.554382    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:20:56.043333 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.043298    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"]
Sep 17 16:20:56.046880 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.046862    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.048940 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.048918    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 17 16:20:56.049036 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.048992    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:20:56.056794 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.056770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"]
Sep 17 16:20:56.168813 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.168777    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.169027 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.168841    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb88370-9722-42d5-a98c-4201aa0f7fad-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.169027 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.168913    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb88370-9722-42d5-a98c-4201aa0f7fad-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.169027 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.168962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbk2s\" (UniqueName: \"kubernetes.io/projected/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-api-access-sbk2s\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.169027 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.169002    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.270069 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.270026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb88370-9722-42d5-a98c-4201aa0f7fad-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.270069 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.270073    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb88370-9722-42d5-a98c-4201aa0f7fad-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.270322 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.270196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbk2s\" (UniqueName: \"kubernetes.io/projected/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-api-access-sbk2s\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.270322 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.270253    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.270322 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.270318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.270541 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.270440    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb88370-9722-42d5-a98c-4201aa0f7fad-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.270925 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.270903    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.271003 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.270923    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.272574 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.272549    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb88370-9722-42d5-a98c-4201aa0f7fad-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.278720 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.278698    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbk2s\" (UniqueName: \"kubernetes.io/projected/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-api-access-sbk2s\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.357755 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.357672    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:20:56.633926 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.633839    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"]
Sep 17 16:20:56.636837 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:20:56.636807    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb88370_9722_42d5_a98c_4201aa0f7fad.slice/crio-acfb14308dfef1bb0a02d8cb1b8508e350b22af8946146992729f6d692e89f08 WatchSource:0}: Error finding container acfb14308dfef1bb0a02d8cb1b8508e350b22af8946146992729f6d692e89f08: Status 404 returned error can't find the container with id acfb14308dfef1bb0a02d8cb1b8508e350b22af8946146992729f6d692e89f08
Sep 17 16:20:56.642995 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:56.642970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" event={"ID":"ecb88370-9722-42d5-a98c-4201aa0f7fad","Type":"ContainerStarted","Data":"acfb14308dfef1bb0a02d8cb1b8508e350b22af8946146992729f6d692e89f08"}
Sep 17 16:20:57.648562 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:20:57.648529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" event={"ID":"ecb88370-9722-42d5-a98c-4201aa0f7fad","Type":"ContainerStarted","Data":"dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834"}
Sep 17 16:21:00.068772 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:00.068741    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"]
Sep 17 16:21:00.069272 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:00.068969    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" containerID="cri-o://39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e" gracePeriod=30
Sep 17 16:21:00.069272 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:00.069009    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kube-rbac-proxy" containerID="cri-o://448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0" gracePeriod=30
Sep 17 16:21:00.553689 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:00.553660    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 17 16:21:00.666454 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:00.666421    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7fdc784-4469-4210-b8af-7ed605a20047" containerID="448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0" exitCode=2
Sep 17 16:21:00.666657 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:00.666511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" event={"ID":"e7fdc784-4469-4210-b8af-7ed605a20047","Type":"ContainerDied","Data":"448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0"}
Sep 17 16:21:01.672703 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:01.672673    2581 generic.go:356] "Generic (PLEG): container finished" podID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerID="dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834" exitCode=0
Sep 17 16:21:01.673072 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:01.672754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" event={"ID":"ecb88370-9722-42d5-a98c-4201aa0f7fad","Type":"ContainerDied","Data":"dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834"}
Sep 17 16:21:02.679672 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:02.679639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" event={"ID":"ecb88370-9722-42d5-a98c-4201aa0f7fad","Type":"ContainerStarted","Data":"a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323"}
Sep 17 16:21:02.680049 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:02.679702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" event={"ID":"ecb88370-9722-42d5-a98c-4201aa0f7fad","Type":"ContainerStarted","Data":"05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1"}
Sep 17 16:21:02.680049 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:02.679884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:21:02.702099 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:02.702051    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" podStartSLOduration=2.7330411039999998 podStartE2EDuration="6.702037133s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.968996029s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:21:02.70176386 +0000 UTC m=+2553.625338426" watchObservedRunningTime="2026-09-17 16:21:02.702037133 +0000 UTC m=+2553.625611699"
Sep 17 16:21:03.259989 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:03.259945    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 17 16:21:03.684544 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:03.684508    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:21:05.211012 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.210976    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"]
Sep 17 16:21:05.214697 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.214672    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.216565 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.216530    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 17 16:21:05.216694 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.216585    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:21:05.225784 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.225763    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"]
Sep 17 16:21:05.360483 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.360432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.360653 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.360523    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cllh4\" (UniqueName: \"kubernetes.io/projected/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-api-access-cllh4\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.360653 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.360547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.360653 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.360576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbfccc1c-1e62-46a7-a559-07ec5bf88648-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.360758 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.360671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.461976 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.461906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.461976 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.461945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cllh4\" (UniqueName: \"kubernetes.io/projected/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-api-access-cllh4\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.461976 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.461964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.462222 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.462194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbfccc1c-1e62-46a7-a559-07ec5bf88648-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.462296 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.462278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.462349 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.462327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.462802 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.462781    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.462853 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.462839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.464631 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.464617    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbfccc1c-1e62-46a7-a559-07ec5bf88648-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.475273 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.475253    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cllh4\" (UniqueName: \"kubernetes.io/projected/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-api-access-cllh4\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.526179 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.526142    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:05.617547 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.617521    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:21:05.697289 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.697254    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7fdc784-4469-4210-b8af-7ed605a20047" containerID="39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e" exitCode=0
Sep 17 16:21:05.697448 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.697313    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"
Sep 17 16:21:05.697448 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.697329    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" event={"ID":"e7fdc784-4469-4210-b8af-7ed605a20047","Type":"ContainerDied","Data":"39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e"}
Sep 17 16:21:05.697448 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.697383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9" event={"ID":"e7fdc784-4469-4210-b8af-7ed605a20047","Type":"ContainerDied","Data":"b010640665abfc050e0a6e5f47b2b9a6ead9d793ae2ceb3613b347014a501bfd"}
Sep 17 16:21:05.697448 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.697410    2581 scope.go:117] "RemoveContainer" containerID="448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0"
Sep 17 16:21:05.704788 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.704774    2581 scope.go:117] "RemoveContainer" containerID="39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e"
Sep 17 16:21:05.711717 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.711703    2581 scope.go:117] "RemoveContainer" containerID="69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157"
Sep 17 16:21:05.718562 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.718544    2581 scope.go:117] "RemoveContainer" containerID="448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0"
Sep 17 16:21:05.718854 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:21:05.718834    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0\": container with ID starting with 448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0 not found: ID does not exist" containerID="448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0"
Sep 17 16:21:05.718944 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.718864    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0"} err="failed to get container status \"448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0\": rpc error: code = NotFound desc = could not find container \"448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0\": container with ID starting with 448aca22b6a6d7c313856da89221e0a5e02b30aaa23ca63cd4827c0c341893f0 not found: ID does not exist"
Sep 17 16:21:05.718944 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.718880    2581 scope.go:117] "RemoveContainer" containerID="39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e"
Sep 17 16:21:05.719119 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:21:05.719101    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e\": container with ID starting with 39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e not found: ID does not exist" containerID="39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e"
Sep 17 16:21:05.719196 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.719124    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e"} err="failed to get container status \"39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e\": rpc error: code = NotFound desc = could not find container \"39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e\": container with ID starting with 39320f56203d7f4333b2d2cad0e2b82dca7ca326f0e912a7eb0f8e173128c86e not found: ID does not exist"
Sep 17 16:21:05.719196 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.719140    2581 scope.go:117] "RemoveContainer" containerID="69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157"
Sep 17 16:21:05.719345 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:21:05.719328    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157\": container with ID starting with 69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157 not found: ID does not exist" containerID="69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157"
Sep 17 16:21:05.719409 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.719347    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157"} err="failed to get container status \"69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157\": rpc error: code = NotFound desc = could not find container \"69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157\": container with ID starting with 69cf0404e4eec4b309a0d2e302f9389126d2157e9cb5cbd0e3a5ebf48fc36157 not found: ID does not exist"
Sep 17 16:21:05.764893 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.764867    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7fdc784-4469-4210-b8af-7ed605a20047-proxy-tls\") pod \"e7fdc784-4469-4210-b8af-7ed605a20047\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") "
Sep 17 16:21:05.765014 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.764903    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5q5v\" (UniqueName: \"kubernetes.io/projected/e7fdc784-4469-4210-b8af-7ed605a20047-kube-api-access-x5q5v\") pod \"e7fdc784-4469-4210-b8af-7ed605a20047\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") "
Sep 17 16:21:05.765014 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.764932    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-kube-rbac-proxy-sar-config\") pod \"e7fdc784-4469-4210-b8af-7ed605a20047\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") "
Sep 17 16:21:05.765105 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.765034    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7fdc784-4469-4210-b8af-7ed605a20047-kserve-provision-location\") pod \"e7fdc784-4469-4210-b8af-7ed605a20047\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") "
Sep 17 16:21:05.765105 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.765085    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-cabundle-cert\") pod \"e7fdc784-4469-4210-b8af-7ed605a20047\" (UID: \"e7fdc784-4469-4210-b8af-7ed605a20047\") "
Sep 17 16:21:05.765316 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.765287    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7fdc784-4469-4210-b8af-7ed605a20047" (UID: "e7fdc784-4469-4210-b8af-7ed605a20047"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:21:05.765438 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.765332    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7fdc784-4469-4210-b8af-7ed605a20047-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7fdc784-4469-4210-b8af-7ed605a20047" (UID: "e7fdc784-4469-4210-b8af-7ed605a20047"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:05.765438 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.765391    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:05.765438 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.765414    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7fdc784-4469-4210-b8af-7ed605a20047" (UID: "e7fdc784-4469-4210-b8af-7ed605a20047"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:21:05.766946 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.766923    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7fdc784-4469-4210-b8af-7ed605a20047-kube-api-access-x5q5v" (OuterVolumeSpecName: "kube-api-access-x5q5v") pod "e7fdc784-4469-4210-b8af-7ed605a20047" (UID: "e7fdc784-4469-4210-b8af-7ed605a20047"). InnerVolumeSpecName "kube-api-access-x5q5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:21:05.767036 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.766949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7fdc784-4469-4210-b8af-7ed605a20047-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7fdc784-4469-4210-b8af-7ed605a20047" (UID: "e7fdc784-4469-4210-b8af-7ed605a20047"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:21:05.811556 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.811534    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"]
Sep 17 16:21:05.813975 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:21:05.813951    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfccc1c_1e62_46a7_a559_07ec5bf88648.slice/crio-32589ecdf5a020daafcbbf0c4d3706a74db1cb1b67e87d1c346160d8353f66a0 WatchSource:0}: Error finding container 32589ecdf5a020daafcbbf0c4d3706a74db1cb1b67e87d1c346160d8353f66a0: Status 404 returned error can't find the container with id 32589ecdf5a020daafcbbf0c4d3706a74db1cb1b67e87d1c346160d8353f66a0
Sep 17 16:21:05.866413 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.866372    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7fdc784-4469-4210-b8af-7ed605a20047-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:05.866413 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.866414    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5q5v\" (UniqueName: \"kubernetes.io/projected/e7fdc784-4469-4210-b8af-7ed605a20047-kube-api-access-x5q5v\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:05.866566 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.866425    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7fdc784-4469-4210-b8af-7ed605a20047-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:05.866566 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:05.866435    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7fdc784-4469-4210-b8af-7ed605a20047-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:06.020568 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:06.020534    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"]
Sep 17 16:21:06.023948 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:06.023923    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-xpjl9"]
Sep 17 16:21:06.702302 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:06.702215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" event={"ID":"dbfccc1c-1e62-46a7-a559-07ec5bf88648","Type":"ContainerStarted","Data":"5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272"}
Sep 17 16:21:06.702302 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:06.702269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" event={"ID":"dbfccc1c-1e62-46a7-a559-07ec5bf88648","Type":"ContainerStarted","Data":"32589ecdf5a020daafcbbf0c4d3706a74db1cb1b67e87d1c346160d8353f66a0"}
Sep 17 16:21:07.558266 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:07.558235    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" path="/var/lib/kubelet/pods/e7fdc784-4469-4210-b8af-7ed605a20047/volumes"
Sep 17 16:21:09.694290 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:09.694258    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:21:10.720671 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:10.720635    2581 generic.go:356] "Generic (PLEG): container finished" podID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerID="5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272" exitCode=0
Sep 17 16:21:10.721010 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:10.720710    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" event={"ID":"dbfccc1c-1e62-46a7-a559-07ec5bf88648","Type":"ContainerDied","Data":"5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272"}
Sep 17 16:21:11.727185 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:11.727151    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" event={"ID":"dbfccc1c-1e62-46a7-a559-07ec5bf88648","Type":"ContainerStarted","Data":"d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111"}
Sep 17 16:21:11.727698 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:11.727210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" event={"ID":"dbfccc1c-1e62-46a7-a559-07ec5bf88648","Type":"ContainerStarted","Data":"a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92"}
Sep 17 16:21:11.727698 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:11.727488    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:11.750758 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:11.750711    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" podStartSLOduration=2.781083312 podStartE2EDuration="6.750692078s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.969608766s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:21:11.74845442 +0000 UTC m=+2562.672028977" watchObservedRunningTime="2026-09-17 16:21:11.750692078 +0000 UTC m=+2562.674266644"
Sep 17 16:21:12.733197 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:12.733164    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:18.743748 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:18.743721    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:39.714976 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:39.714934    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 17 16:21:48.748191 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:48.748157    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:49.697327 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:49.697292    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:21:55.293321 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:55.293289    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"]
Sep 17 16:21:55.293882 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:55.293541    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="kserve-container" containerID="cri-o://a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92" gracePeriod=30
Sep 17 16:21:55.293882 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:55.293617    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="kube-rbac-proxy" containerID="cri-o://d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111" gracePeriod=30
Sep 17 16:21:55.922127 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:55.922089    2581 generic.go:356] "Generic (PLEG): container finished" podID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerID="d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111" exitCode=2
Sep 17 16:21:55.922310 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:55.922155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" event={"ID":"dbfccc1c-1e62-46a7-a559-07ec5bf88648","Type":"ContainerDied","Data":"d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111"}
Sep 17 16:21:56.057230 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.057194    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"]
Sep 17 16:21:56.057492 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.057446    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kserve-container" containerID="cri-o://05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1" gracePeriod=30
Sep 17 16:21:56.057643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.057508    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kube-rbac-proxy" containerID="cri-o://a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323" gracePeriod=30
Sep 17 16:21:56.333362 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.333332    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:56.428624 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.428590    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbfccc1c-1e62-46a7-a559-07ec5bf88648-proxy-tls\") pod \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") "
Sep 17 16:21:56.428788 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.428636    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kserve-provision-location\") pod \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") "
Sep 17 16:21:56.428788 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.428658    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-cabundle-cert\") pod \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") "
Sep 17 16:21:56.428788 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.428689    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-rbac-proxy-sar-config\") pod \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") "
Sep 17 16:21:56.428788 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.428717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cllh4\" (UniqueName: \"kubernetes.io/projected/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-api-access-cllh4\") pod \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\" (UID: \"dbfccc1c-1e62-46a7-a559-07ec5bf88648\") "
Sep 17 16:21:56.429001 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.428971    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dbfccc1c-1e62-46a7-a559-07ec5bf88648" (UID: "dbfccc1c-1e62-46a7-a559-07ec5bf88648"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:21:56.429079 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.429057    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dbfccc1c-1e62-46a7-a559-07ec5bf88648" (UID: "dbfccc1c-1e62-46a7-a559-07ec5bf88648"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:21:56.429141 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.429062    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dbfccc1c-1e62-46a7-a559-07ec5bf88648" (UID: "dbfccc1c-1e62-46a7-a559-07ec5bf88648"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:21:56.430809 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.430791    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbfccc1c-1e62-46a7-a559-07ec5bf88648-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dbfccc1c-1e62-46a7-a559-07ec5bf88648" (UID: "dbfccc1c-1e62-46a7-a559-07ec5bf88648"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:21:56.430898 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.430844    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-api-access-cllh4" (OuterVolumeSpecName: "kube-api-access-cllh4") pod "dbfccc1c-1e62-46a7-a559-07ec5bf88648" (UID: "dbfccc1c-1e62-46a7-a559-07ec5bf88648"). InnerVolumeSpecName "kube-api-access-cllh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:21:56.529609 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.529529    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:56.529609 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.529556    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cllh4\" (UniqueName: \"kubernetes.io/projected/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kube-api-access-cllh4\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:56.529609 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.529565    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dbfccc1c-1e62-46a7-a559-07ec5bf88648-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:56.529609 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.529575    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dbfccc1c-1e62-46a7-a559-07ec5bf88648-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:56.529609 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.529585    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dbfccc1c-1e62-46a7-a559-07ec5bf88648-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:21:56.928508 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.928457    2581 generic.go:356] "Generic (PLEG): container finished" podID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerID="a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323" exitCode=2
Sep 17 16:21:56.928508 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.928498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" event={"ID":"ecb88370-9722-42d5-a98c-4201aa0f7fad","Type":"ContainerDied","Data":"a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323"}
Sep 17 16:21:56.930292 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.930270    2581 generic.go:356] "Generic (PLEG): container finished" podID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerID="a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92" exitCode=0
Sep 17 16:21:56.930408 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.930317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" event={"ID":"dbfccc1c-1e62-46a7-a559-07ec5bf88648","Type":"ContainerDied","Data":"a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92"}
Sep 17 16:21:56.930408 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.930336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"
Sep 17 16:21:56.930408 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.930368    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w" event={"ID":"dbfccc1c-1e62-46a7-a559-07ec5bf88648","Type":"ContainerDied","Data":"32589ecdf5a020daafcbbf0c4d3706a74db1cb1b67e87d1c346160d8353f66a0"}
Sep 17 16:21:56.930408 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.930394    2581 scope.go:117] "RemoveContainer" containerID="d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111"
Sep 17 16:21:56.938638 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.938602    2581 scope.go:117] "RemoveContainer" containerID="a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92"
Sep 17 16:21:56.945995 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.945979    2581 scope.go:117] "RemoveContainer" containerID="5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272"
Sep 17 16:21:56.953750 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.953718    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"]
Sep 17 16:21:56.954078 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.954057    2581 scope.go:117] "RemoveContainer" containerID="d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111"
Sep 17 16:21:56.954437 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:21:56.954408    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111\": container with ID starting with d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111 not found: ID does not exist" containerID="d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111"
Sep 17 16:21:56.954530 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.954448    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111"} err="failed to get container status \"d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111\": rpc error: code = NotFound desc = could not find container \"d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111\": container with ID starting with d6931f70d076e25ed3bf5a8e7eb82f37f407711856cc7b9991108181b548d111 not found: ID does not exist"
Sep 17 16:21:56.954530 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.954487    2581 scope.go:117] "RemoveContainer" containerID="a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92"
Sep 17 16:21:56.954826 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:21:56.954799    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92\": container with ID starting with a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92 not found: ID does not exist" containerID="a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92"
Sep 17 16:21:56.954996 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.954834    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92"} err="failed to get container status \"a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92\": rpc error: code = NotFound desc = could not find container \"a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92\": container with ID starting with a0e164037d2ddcb70039c3f7e3d7788615ee0499399bb5960f6bc03a7dd60a92 not found: ID does not exist"
Sep 17 16:21:56.954996 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.954854    2581 scope.go:117] "RemoveContainer" containerID="5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272"
Sep 17 16:21:56.955195 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:21:56.955173    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272\": container with ID starting with 5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272 not found: ID does not exist" containerID="5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272"
Sep 17 16:21:56.955257 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.955200    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272"} err="failed to get container status \"5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272\": rpc error: code = NotFound desc = could not find container \"5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272\": container with ID starting with 5e0f80fccaa34eabf498bfaba74945e775d0a1aa88c975982c45631bf903c272 not found: ID does not exist"
Sep 17 16:21:56.956374 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:56.956359    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-cmv7w"]
Sep 17 16:21:57.559335 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:57.559297    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" path="/var/lib/kubelet/pods/dbfccc1c-1e62-46a7-a559-07ec5bf88648/volumes"
Sep 17 16:21:59.690253 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:59.690210    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 17 16:21:59.695730 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:21:59.695699    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.50:8080/v2/models/isvc-xgboost-v2-runtime/ready\": dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 17 16:22:01.235497 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235449    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"]
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235771    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="kube-rbac-proxy"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235782    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="kube-rbac-proxy"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235794    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235799    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235812    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kube-rbac-proxy"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235818    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kube-rbac-proxy"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235827    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="kserve-container"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235832    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="kserve-container"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235843    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="storage-initializer"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235847    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="storage-initializer"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235854    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="storage-initializer"
Sep 17 16:22:01.235894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235858    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="storage-initializer"
Sep 17 16:22:01.236249 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235904    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kserve-container"
Sep 17 16:22:01.236249 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235912    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="kube-rbac-proxy"
Sep 17 16:22:01.236249 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235920    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7fdc784-4469-4210-b8af-7ed605a20047" containerName="kube-rbac-proxy"
Sep 17 16:22:01.236249 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.235926    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbfccc1c-1e62-46a7-a559-07ec5bf88648" containerName="kserve-container"
Sep 17 16:22:01.240859 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.240840    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.242645 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.242622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 16:22:01.242735 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.242628    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 16:22:01.249507 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.249487    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"]
Sep 17 16:22:01.270878 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.270854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71657e22-51cc-42e4-b257-dbd1d387d280-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.270986 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.270889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.270986 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.270922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71657e22-51cc-42e4-b257-dbd1d387d280-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.271082 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.270991    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6vgq\" (UniqueName: \"kubernetes.io/projected/71657e22-51cc-42e4-b257-dbd1d387d280-kube-api-access-m6vgq\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.271082 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.271044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.372181 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.372143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71657e22-51cc-42e4-b257-dbd1d387d280-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.372181 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.372183    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6vgq\" (UniqueName: \"kubernetes.io/projected/71657e22-51cc-42e4-b257-dbd1d387d280-kube-api-access-m6vgq\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.372422 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.372223    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.372422 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.372249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71657e22-51cc-42e4-b257-dbd1d387d280-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.372422 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.372281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.372422 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:22:01.372293    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 16:22:01.372659 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:22:01.372445    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/71657e22-51cc-42e4-b257-dbd1d387d280-proxy-tls podName:71657e22-51cc-42e4-b257-dbd1d387d280 nodeName:}" failed. No retries permitted until 2026-09-17 16:22:01.872414092 +0000 UTC m=+2612.795988651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/71657e22-51cc-42e4-b257-dbd1d387d280-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-tdbqd" (UID: "71657e22-51cc-42e4-b257-dbd1d387d280") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 17 16:22:01.372766 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.372746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71657e22-51cc-42e4-b257-dbd1d387d280-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.373011 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.372993    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.373048 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.373027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.383039 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.383015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6vgq\" (UniqueName: \"kubernetes.io/projected/71657e22-51cc-42e4-b257-dbd1d387d280-kube-api-access-m6vgq\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.877481 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.877435    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71657e22-51cc-42e4-b257-dbd1d387d280-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:01.879873 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:01.879837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71657e22-51cc-42e4-b257-dbd1d387d280-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tdbqd\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:02.151532 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:02.151418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:02.425021 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:02.424998    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"]
Sep 17 16:22:02.427619 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:22:02.427594    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71657e22_51cc_42e4_b257_dbd1d387d280.slice/crio-58f87edeaa0f57479a0a8d428a1756ccd8ddc43f3e9f8cda666c85981ceb5031 WatchSource:0}: Error finding container 58f87edeaa0f57479a0a8d428a1756ccd8ddc43f3e9f8cda666c85981ceb5031: Status 404 returned error can't find the container with id 58f87edeaa0f57479a0a8d428a1756ccd8ddc43f3e9f8cda666c85981ceb5031
Sep 17 16:22:02.957313 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:02.957273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" event={"ID":"71657e22-51cc-42e4-b257-dbd1d387d280","Type":"ContainerStarted","Data":"09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93"}
Sep 17 16:22:02.957513 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:02.957365    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" event={"ID":"71657e22-51cc-42e4-b257-dbd1d387d280","Type":"ContainerStarted","Data":"58f87edeaa0f57479a0a8d428a1756ccd8ddc43f3e9f8cda666c85981ceb5031"}
Sep 17 16:22:03.593262 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.593241    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:22:03.693252 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.693159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb88370-9722-42d5-a98c-4201aa0f7fad-proxy-tls\") pod \"ecb88370-9722-42d5-a98c-4201aa0f7fad\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") "
Sep 17 16:22:03.693252 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.693198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbk2s\" (UniqueName: \"kubernetes.io/projected/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-api-access-sbk2s\") pod \"ecb88370-9722-42d5-a98c-4201aa0f7fad\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") "
Sep 17 16:22:03.693252 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.693217    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-rbac-proxy-sar-config\") pod \"ecb88370-9722-42d5-a98c-4201aa0f7fad\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") "
Sep 17 16:22:03.693252 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.693254    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-cabundle-cert\") pod \"ecb88370-9722-42d5-a98c-4201aa0f7fad\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") "
Sep 17 16:22:03.693618 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.693274    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb88370-9722-42d5-a98c-4201aa0f7fad-kserve-provision-location\") pod \"ecb88370-9722-42d5-a98c-4201aa0f7fad\" (UID: \"ecb88370-9722-42d5-a98c-4201aa0f7fad\") "
Sep 17 16:22:03.693660 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.693606    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecb88370-9722-42d5-a98c-4201aa0f7fad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ecb88370-9722-42d5-a98c-4201aa0f7fad" (UID: "ecb88370-9722-42d5-a98c-4201aa0f7fad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:22:03.693660 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.693645    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ecb88370-9722-42d5-a98c-4201aa0f7fad" (UID: "ecb88370-9722-42d5-a98c-4201aa0f7fad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:22:03.693726 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.693660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ecb88370-9722-42d5-a98c-4201aa0f7fad" (UID: "ecb88370-9722-42d5-a98c-4201aa0f7fad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:22:03.695300 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.695278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecb88370-9722-42d5-a98c-4201aa0f7fad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ecb88370-9722-42d5-a98c-4201aa0f7fad" (UID: "ecb88370-9722-42d5-a98c-4201aa0f7fad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:22:03.695384 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.695360    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-api-access-sbk2s" (OuterVolumeSpecName: "kube-api-access-sbk2s") pod "ecb88370-9722-42d5-a98c-4201aa0f7fad" (UID: "ecb88370-9722-42d5-a98c-4201aa0f7fad"). InnerVolumeSpecName "kube-api-access-sbk2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:22:03.794084 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.794041    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ecb88370-9722-42d5-a98c-4201aa0f7fad-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:22:03.794084 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.794071    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbk2s\" (UniqueName: \"kubernetes.io/projected/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-api-access-sbk2s\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:22:03.794084 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.794080    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:22:03.794084 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.794089    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ecb88370-9722-42d5-a98c-4201aa0f7fad-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:22:03.794084 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.794097    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ecb88370-9722-42d5-a98c-4201aa0f7fad-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:22:03.963240 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.963149    2581 generic.go:356] "Generic (PLEG): container finished" podID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerID="05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1" exitCode=0
Sep 17 16:22:03.963240 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.963204    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"
Sep 17 16:22:03.963418 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.963235    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" event={"ID":"ecb88370-9722-42d5-a98c-4201aa0f7fad","Type":"ContainerDied","Data":"05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1"}
Sep 17 16:22:03.963418 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.963294    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk" event={"ID":"ecb88370-9722-42d5-a98c-4201aa0f7fad","Type":"ContainerDied","Data":"acfb14308dfef1bb0a02d8cb1b8508e350b22af8946146992729f6d692e89f08"}
Sep 17 16:22:03.963418 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.963323    2581 scope.go:117] "RemoveContainer" containerID="a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323"
Sep 17 16:22:03.973343 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.973324    2581 scope.go:117] "RemoveContainer" containerID="05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1"
Sep 17 16:22:03.980555 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.980538    2581 scope.go:117] "RemoveContainer" containerID="dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834"
Sep 17 16:22:03.987413 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.987398    2581 scope.go:117] "RemoveContainer" containerID="a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323"
Sep 17 16:22:03.987686 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:22:03.987669    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323\": container with ID starting with a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323 not found: ID does not exist" containerID="a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323"
Sep 17 16:22:03.987762 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.987699    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323"} err="failed to get container status \"a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323\": rpc error: code = NotFound desc = could not find container \"a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323\": container with ID starting with a016e5063037eaecf5e349732432d0e64599fee2064155d86ca52d95e90f4323 not found: ID does not exist"
Sep 17 16:22:03.987762 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.987738    2581 scope.go:117] "RemoveContainer" containerID="05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1"
Sep 17 16:22:03.988027 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:22:03.988010    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1\": container with ID starting with 05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1 not found: ID does not exist" containerID="05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1"
Sep 17 16:22:03.988067 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.988034    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1"} err="failed to get container status \"05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1\": rpc error: code = NotFound desc = could not find container \"05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1\": container with ID starting with 05fef283ba3f0501f743db3ce84674a7c61c2662907ad6d1f023c1402cde8dd1 not found: ID does not exist"
Sep 17 16:22:03.988067 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.988048    2581 scope.go:117] "RemoveContainer" containerID="dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834"
Sep 17 16:22:03.988257 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:22:03.988240    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834\": container with ID starting with dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834 not found: ID does not exist" containerID="dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834"
Sep 17 16:22:03.988300 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.988265    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834"} err="failed to get container status \"dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834\": rpc error: code = NotFound desc = could not find container \"dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834\": container with ID starting with dc32cd28c753a4363152aed1eaa19ce80a00848e529ae7e28f10efc6d9090834 not found: ID does not exist"
Sep 17 16:22:03.993531 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:03.993509    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"]
Sep 17 16:22:04.001710 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:04.001690    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-45gtk"]
Sep 17 16:22:05.557319 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:05.557286    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" path="/var/lib/kubelet/pods/ecb88370-9722-42d5-a98c-4201aa0f7fad/volumes"
Sep 17 16:22:06.977694 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:06.977663    2581 generic.go:356] "Generic (PLEG): container finished" podID="71657e22-51cc-42e4-b257-dbd1d387d280" containerID="09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93" exitCode=0
Sep 17 16:22:06.978079 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:06.977731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" event={"ID":"71657e22-51cc-42e4-b257-dbd1d387d280","Type":"ContainerDied","Data":"09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93"}
Sep 17 16:22:07.983417 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:07.983383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" event={"ID":"71657e22-51cc-42e4-b257-dbd1d387d280","Type":"ContainerStarted","Data":"d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805"}
Sep 17 16:22:07.983920 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:07.983500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" event={"ID":"71657e22-51cc-42e4-b257-dbd1d387d280","Type":"ContainerStarted","Data":"0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068"}
Sep 17 16:22:07.983920 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:07.983726    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:08.001217 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:08.001166    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podStartSLOduration=3.143172742 podStartE2EDuration="7.001153363s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.857980621s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:22:07.999903569 +0000 UTC m=+2618.923478135" watchObservedRunningTime="2026-09-17 16:22:08.001153363 +0000 UTC m=+2618.924727929"
Sep 17 16:22:08.987674 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:08.987647    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:08.989023 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:08.988994    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:22:09.991900 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:09.991864    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:22:14.996346 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:14.996317    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:22:14.996827 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:14.996798    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:22:24.997129 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:24.997093    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:22:34.997808 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:34.997765    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:22:35.806747 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.806711    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"]
Sep 17 16:22:35.807124 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.807105    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kube-rbac-proxy"
Sep 17 16:22:35.807213 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.807127    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kube-rbac-proxy"
Sep 17 16:22:35.807213 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.807162    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="storage-initializer"
Sep 17 16:22:35.807213 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.807171    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="storage-initializer"
Sep 17 16:22:35.807213 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.807182    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kserve-container"
Sep 17 16:22:35.807213 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.807191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kserve-container"
Sep 17 16:22:35.807494 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.807284    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kube-rbac-proxy"
Sep 17 16:22:35.807494 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.807301    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecb88370-9722-42d5-a98c-4201aa0f7fad" containerName="kserve-container"
Sep 17 16:22:35.812085 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.812063    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:35.814024 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.813995    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 17 16:22:35.814156 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.814047    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:22:35.820871 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.820845    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"]
Sep 17 16:22:35.968365 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.968329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdcnf\" (UniqueName: \"kubernetes.io/projected/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-api-access-zdcnf\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:35.968365 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.968367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:35.968656 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.968432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:35.968656 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.968498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:35.968656 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:35.968526    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.069850 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.069756    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdcnf\" (UniqueName: \"kubernetes.io/projected/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-api-access-zdcnf\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.069850 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.069797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.069850 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.069824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.069850 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.069840    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.070568 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:22:36.069975    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 16:22:36.070568 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:22:36.070080    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-proxy-tls podName:6e1bbe34-fafb-457a-89b3-6964fa89cfa1 nodeName:}" failed. No retries permitted until 2026-09-17 16:22:36.570064383 +0000 UTC m=+2647.493638941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" (UID: "6e1bbe34-fafb-457a-89b3-6964fa89cfa1") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 17 16:22:36.070568 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.070221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.070568 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.070397    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.070568 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.070532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.070974 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.070954    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.080639 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.080620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdcnf\" (UniqueName: \"kubernetes.io/projected/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-api-access-zdcnf\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.575105 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.575071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.577565 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.577542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.723606 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.723569    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:36.998011 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:36.997982    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"]
Sep 17 16:22:37.000600 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:22:37.000572    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1bbe34_fafb_457a_89b3_6964fa89cfa1.slice/crio-e5f919d680ac138e6912ea8d19a01be307c3b7d02dcaf950a44545fc546ad06a WatchSource:0}: Error finding container e5f919d680ac138e6912ea8d19a01be307c3b7d02dcaf950a44545fc546ad06a: Status 404 returned error can't find the container with id e5f919d680ac138e6912ea8d19a01be307c3b7d02dcaf950a44545fc546ad06a
Sep 17 16:22:37.108838 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:37.108805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" event={"ID":"6e1bbe34-fafb-457a-89b3-6964fa89cfa1","Type":"ContainerStarted","Data":"7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477"}
Sep 17 16:22:37.109214 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:37.108867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" event={"ID":"6e1bbe34-fafb-457a-89b3-6964fa89cfa1","Type":"ContainerStarted","Data":"e5f919d680ac138e6912ea8d19a01be307c3b7d02dcaf950a44545fc546ad06a"}
Sep 17 16:22:41.126508 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:41.126474    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerID="7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477" exitCode=0
Sep 17 16:22:41.126877 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:41.126533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" event={"ID":"6e1bbe34-fafb-457a-89b3-6964fa89cfa1","Type":"ContainerDied","Data":"7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477"}
Sep 17 16:22:42.134690 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:42.134652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" event={"ID":"6e1bbe34-fafb-457a-89b3-6964fa89cfa1","Type":"ContainerStarted","Data":"271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b"}
Sep 17 16:22:42.135185 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:42.134715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" event={"ID":"6e1bbe34-fafb-457a-89b3-6964fa89cfa1","Type":"ContainerStarted","Data":"92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7"}
Sep 17 16:22:42.135185 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:42.134911    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:42.155596 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:42.155532    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" podStartSLOduration=3.203476485 podStartE2EDuration="7.155511333s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.952034848s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:22:42.153889407 +0000 UTC m=+2653.077463974" watchObservedRunningTime="2026-09-17 16:22:42.155511333 +0000 UTC m=+2653.079085901"
Sep 17 16:22:43.139450 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:43.139410    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:44.996780 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:44.996736    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:22:49.151205 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:49.151175    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:22:54.997586 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:22:54.997543    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:23:04.997728 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:04.997686    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:23:14.997617 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:14.997588    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:23:19.155448 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:19.155417    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:23:21.902113 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:21.902030    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"]
Sep 17 16:23:21.902540 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:21.902281    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" containerID="cri-o://0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068" gracePeriod=30
Sep 17 16:23:21.902540 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:21.902320    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kube-rbac-proxy" containerID="cri-o://d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805" gracePeriod=30
Sep 17 16:23:22.320418 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:22.320380    2581 generic.go:356] "Generic (PLEG): container finished" podID="71657e22-51cc-42e4-b257-dbd1d387d280" containerID="d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805" exitCode=2
Sep 17 16:23:22.320615 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:22.320456    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" event={"ID":"71657e22-51cc-42e4-b257-dbd1d387d280","Type":"ContainerDied","Data":"d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805"}
Sep 17 16:23:24.992366 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:24.992319    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 17 16:23:24.997869 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:24.997830    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 17 16:23:25.927067 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:25.927032    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"]
Sep 17 16:23:25.927272 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:25.927250    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="kserve-container" containerID="cri-o://92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7" gracePeriod=30
Sep 17 16:23:25.927346 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:25.927304    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="kube-rbac-proxy" containerID="cri-o://271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b" gracePeriod=30
Sep 17 16:23:26.251401 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.251375    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:23:26.340445 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.340405    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerID="271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b" exitCode=2
Sep 17 16:23:26.340639 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.340490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" event={"ID":"6e1bbe34-fafb-457a-89b3-6964fa89cfa1","Type":"ContainerDied","Data":"271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b"}
Sep 17 16:23:26.342602 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.342574    2581 generic.go:356] "Generic (PLEG): container finished" podID="71657e22-51cc-42e4-b257-dbd1d387d280" containerID="0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068" exitCode=0
Sep 17 16:23:26.342749 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.342640    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"
Sep 17 16:23:26.342749 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.342663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" event={"ID":"71657e22-51cc-42e4-b257-dbd1d387d280","Type":"ContainerDied","Data":"0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068"}
Sep 17 16:23:26.342749 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.342737    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd" event={"ID":"71657e22-51cc-42e4-b257-dbd1d387d280","Type":"ContainerDied","Data":"58f87edeaa0f57479a0a8d428a1756ccd8ddc43f3e9f8cda666c85981ceb5031"}
Sep 17 16:23:26.342924 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.342775    2581 scope.go:117] "RemoveContainer" containerID="d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805"
Sep 17 16:23:26.351179 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.351158    2581 scope.go:117] "RemoveContainer" containerID="0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068"
Sep 17 16:23:26.359329 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.359308    2581 scope.go:117] "RemoveContainer" containerID="09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93"
Sep 17 16:23:26.367397 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.367376    2581 scope.go:117] "RemoveContainer" containerID="d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805"
Sep 17 16:23:26.367737 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:23:26.367713    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805\": container with ID starting with d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805 not found: ID does not exist" containerID="d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805"
Sep 17 16:23:26.367815 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.367748    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805"} err="failed to get container status \"d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805\": rpc error: code = NotFound desc = could not find container \"d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805\": container with ID starting with d9bca98f51da3259c022fe1321e6f5b513a0683b4b28a558f9d9619f9f4c4805 not found: ID does not exist"
Sep 17 16:23:26.367815 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.367766    2581 scope.go:117] "RemoveContainer" containerID="0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068"
Sep 17 16:23:26.368109 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:23:26.368086    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068\": container with ID starting with 0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068 not found: ID does not exist" containerID="0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068"
Sep 17 16:23:26.368150 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.368121    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068"} err="failed to get container status \"0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068\": rpc error: code = NotFound desc = could not find container \"0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068\": container with ID starting with 0a706d7784f3bf667b83619754d342fd249d59acb14795e4c4ac63b6f3b40068 not found: ID does not exist"
Sep 17 16:23:26.368150 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.368142    2581 scope.go:117] "RemoveContainer" containerID="09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93"
Sep 17 16:23:26.368425 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:23:26.368400    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93\": container with ID starting with 09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93 not found: ID does not exist" containerID="09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93"
Sep 17 16:23:26.368540 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.368427    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93"} err="failed to get container status \"09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93\": rpc error: code = NotFound desc = could not find container \"09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93\": container with ID starting with 09d9d56d39933cc9dc8aff6f49e697d41cef32da8b3be7b08664b2cf19aede93 not found: ID does not exist"
Sep 17 16:23:26.410404 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.410357    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71657e22-51cc-42e4-b257-dbd1d387d280-proxy-tls\") pod \"71657e22-51cc-42e4-b257-dbd1d387d280\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") "
Sep 17 16:23:26.410620 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.410437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6vgq\" (UniqueName: \"kubernetes.io/projected/71657e22-51cc-42e4-b257-dbd1d387d280-kube-api-access-m6vgq\") pod \"71657e22-51cc-42e4-b257-dbd1d387d280\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") "
Sep 17 16:23:26.410620 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.410499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71657e22-51cc-42e4-b257-dbd1d387d280-kserve-provision-location\") pod \"71657e22-51cc-42e4-b257-dbd1d387d280\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") "
Sep 17 16:23:26.410620 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.410517    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-cabundle-cert\") pod \"71657e22-51cc-42e4-b257-dbd1d387d280\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") "
Sep 17 16:23:26.410620 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.410546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-kube-rbac-proxy-sar-config\") pod \"71657e22-51cc-42e4-b257-dbd1d387d280\" (UID: \"71657e22-51cc-42e4-b257-dbd1d387d280\") "
Sep 17 16:23:26.410919 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.410894    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71657e22-51cc-42e4-b257-dbd1d387d280-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "71657e22-51cc-42e4-b257-dbd1d387d280" (UID: "71657e22-51cc-42e4-b257-dbd1d387d280"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:23:26.410999 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.410930    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "71657e22-51cc-42e4-b257-dbd1d387d280" (UID: "71657e22-51cc-42e4-b257-dbd1d387d280"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:23:26.410999 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.410947    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "71657e22-51cc-42e4-b257-dbd1d387d280" (UID: "71657e22-51cc-42e4-b257-dbd1d387d280"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:23:26.412609 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.412587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71657e22-51cc-42e4-b257-dbd1d387d280-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "71657e22-51cc-42e4-b257-dbd1d387d280" (UID: "71657e22-51cc-42e4-b257-dbd1d387d280"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:23:26.412918 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.412890    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71657e22-51cc-42e4-b257-dbd1d387d280-kube-api-access-m6vgq" (OuterVolumeSpecName: "kube-api-access-m6vgq") pod "71657e22-51cc-42e4-b257-dbd1d387d280" (UID: "71657e22-51cc-42e4-b257-dbd1d387d280"). InnerVolumeSpecName "kube-api-access-m6vgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:23:26.511397 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.511305    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/71657e22-51cc-42e4-b257-dbd1d387d280-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:26.511397 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.511338    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6vgq\" (UniqueName: \"kubernetes.io/projected/71657e22-51cc-42e4-b257-dbd1d387d280-kube-api-access-m6vgq\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:26.511397 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.511348    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/71657e22-51cc-42e4-b257-dbd1d387d280-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:26.511397 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.511357    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:26.511397 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.511367    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/71657e22-51cc-42e4-b257-dbd1d387d280-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:26.668830 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.668795    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"]
Sep 17 16:23:26.678856 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:26.678827    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tdbqd"]
Sep 17 16:23:27.294911 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.294884    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:23:27.349636 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.349603    2581 generic.go:356] "Generic (PLEG): container finished" podID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerID="92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7" exitCode=0
Sep 17 16:23:27.349805 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.349662    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"
Sep 17 16:23:27.349805 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.349661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" event={"ID":"6e1bbe34-fafb-457a-89b3-6964fa89cfa1","Type":"ContainerDied","Data":"92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7"}
Sep 17 16:23:27.349805 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.349788    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2" event={"ID":"6e1bbe34-fafb-457a-89b3-6964fa89cfa1","Type":"ContainerDied","Data":"e5f919d680ac138e6912ea8d19a01be307c3b7d02dcaf950a44545fc546ad06a"}
Sep 17 16:23:27.349922 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.349815    2581 scope.go:117] "RemoveContainer" containerID="271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b"
Sep 17 16:23:27.358136 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.358118    2581 scope.go:117] "RemoveContainer" containerID="92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7"
Sep 17 16:23:27.366059 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.366040    2581 scope.go:117] "RemoveContainer" containerID="7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477"
Sep 17 16:23:27.373694 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.373675    2581 scope.go:117] "RemoveContainer" containerID="271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b"
Sep 17 16:23:27.374005 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:23:27.373986    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b\": container with ID starting with 271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b not found: ID does not exist" containerID="271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b"
Sep 17 16:23:27.374054 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.374023    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b"} err="failed to get container status \"271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b\": rpc error: code = NotFound desc = could not find container \"271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b\": container with ID starting with 271378fcedb9ec216202bb80a6888b61a4ce8f4e93d0e9356363a35ce8150e8b not found: ID does not exist"
Sep 17 16:23:27.374054 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.374043    2581 scope.go:117] "RemoveContainer" containerID="92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7"
Sep 17 16:23:27.374314 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:23:27.374297    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7\": container with ID starting with 92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7 not found: ID does not exist" containerID="92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7"
Sep 17 16:23:27.374387 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.374318    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7"} err="failed to get container status \"92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7\": rpc error: code = NotFound desc = could not find container \"92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7\": container with ID starting with 92ccb05eeef9f69975fe1185fabe588bbb783bc938460becbc7d9766236fcfa7 not found: ID does not exist"
Sep 17 16:23:27.374387 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.374334    2581 scope.go:117] "RemoveContainer" containerID="7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477"
Sep 17 16:23:27.374611 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:23:27.374591    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477\": container with ID starting with 7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477 not found: ID does not exist" containerID="7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477"
Sep 17 16:23:27.374685 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.374616    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477"} err="failed to get container status \"7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477\": rpc error: code = NotFound desc = could not find container \"7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477\": container with ID starting with 7b9e26a13ace597247f09f8aa19215304c48c58c318a95021770c170597a0477 not found: ID does not exist"
Sep 17 16:23:27.421499 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.421453    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kserve-provision-location\") pod \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") "
Sep 17 16:23:27.421618 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.421506    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-rbac-proxy-sar-config\") pod \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") "
Sep 17 16:23:27.421618 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.421550    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-proxy-tls\") pod \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") "
Sep 17 16:23:27.421618 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.421575    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdcnf\" (UniqueName: \"kubernetes.io/projected/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-api-access-zdcnf\") pod \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") "
Sep 17 16:23:27.421731 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.421672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-cabundle-cert\") pod \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\" (UID: \"6e1bbe34-fafb-457a-89b3-6964fa89cfa1\") "
Sep 17 16:23:27.421888 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.421856    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e1bbe34-fafb-457a-89b3-6964fa89cfa1" (UID: "6e1bbe34-fafb-457a-89b3-6964fa89cfa1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:23:27.422026 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.421923    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e1bbe34-fafb-457a-89b3-6964fa89cfa1" (UID: "6e1bbe34-fafb-457a-89b3-6964fa89cfa1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:23:27.422142 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.422124    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e1bbe34-fafb-457a-89b3-6964fa89cfa1" (UID: "6e1bbe34-fafb-457a-89b3-6964fa89cfa1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:23:27.423813 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.423792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-api-access-zdcnf" (OuterVolumeSpecName: "kube-api-access-zdcnf") pod "6e1bbe34-fafb-457a-89b3-6964fa89cfa1" (UID: "6e1bbe34-fafb-457a-89b3-6964fa89cfa1"). InnerVolumeSpecName "kube-api-access-zdcnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:23:27.423888 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.423818    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e1bbe34-fafb-457a-89b3-6964fa89cfa1" (UID: "6e1bbe34-fafb-457a-89b3-6964fa89cfa1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:23:27.522704 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.522653    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:27.522704 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.522700    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdcnf\" (UniqueName: \"kubernetes.io/projected/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-api-access-zdcnf\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:27.522704 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.522714    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:27.522941 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.522727    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:27.522941 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.522739    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e1bbe34-fafb-457a-89b3-6964fa89cfa1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:23:27.557247 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.557215    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" path="/var/lib/kubelet/pods/71657e22-51cc-42e4-b257-dbd1d387d280/volumes"
Sep 17 16:23:27.672353 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.669250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"]
Sep 17 16:23:27.674805 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:27.674777    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-xgkt2"]
Sep 17 16:23:29.557261 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:23:29.557219    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" path="/var/lib/kubelet/pods/6e1bbe34-fafb-457a-89b3-6964fa89cfa1/volumes"
Sep 17 16:24:56.381674 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.381588    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"]
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382050    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="kserve-container"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382067    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="kserve-container"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382084    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382093    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382106    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kube-rbac-proxy"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382113    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kube-rbac-proxy"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382124    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="storage-initializer"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382129    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="storage-initializer"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382142    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="storage-initializer"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382147    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="storage-initializer"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382156    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="kube-rbac-proxy"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382160    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="kube-rbac-proxy"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382210    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="kserve-container"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382217    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e1bbe34-fafb-457a-89b3-6964fa89cfa1" containerName="kube-rbac-proxy"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382222    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kube-rbac-proxy"
Sep 17 16:24:56.384308 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.382229    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="71657e22-51cc-42e4-b257-dbd1d387d280" containerName="kserve-container"
Sep 17 16:24:56.385266 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.385247    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.387354 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.387327    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 17 16:24:56.387487 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.387431    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:24:56.387487 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.387440    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 17 16:24:56.387487 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.387436    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:24:56.387709 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.387691    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:24:56.387784 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.387755    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:24:56.387894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.387877    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:24:56.397282 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.397256    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"]
Sep 17 16:24:56.483654 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.483612    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88c5538b-7a58-4ccc-ad85-bd72543446eb-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.483836 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.483663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j26ft\" (UniqueName: \"kubernetes.io/projected/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-api-access-j26ft\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.483836 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.483687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.483836 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.483761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88c5538b-7a58-4ccc-ad85-bd72543446eb-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.483836 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.483818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.584612 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.584575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j26ft\" (UniqueName: \"kubernetes.io/projected/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-api-access-j26ft\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.584612 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.584611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.584861 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.584654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88c5538b-7a58-4ccc-ad85-bd72543446eb-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.584861 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.584709    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.584861 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.584773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88c5538b-7a58-4ccc-ad85-bd72543446eb-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.585137 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.585114    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88c5538b-7a58-4ccc-ad85-bd72543446eb-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.585479 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.585441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.585542 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.585444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.587142 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.587120    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88c5538b-7a58-4ccc-ad85-bd72543446eb-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.594597 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.594572    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j26ft\" (UniqueName: \"kubernetes.io/projected/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-api-access-j26ft\") pod \"isvc-paddle-predictor-6d5c9fd4f6-pzrq5\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.695861 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.695782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:24:56.971526 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:56.971398    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"]
Sep 17 16:24:56.974902 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:24:56.974874    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c5538b_7a58_4ccc_ad85_bd72543446eb.slice/crio-fba28d6fd77b7dc4e8263b3678873d36015c510fdc9df5c810adb309e8067a07 WatchSource:0}: Error finding container fba28d6fd77b7dc4e8263b3678873d36015c510fdc9df5c810adb309e8067a07: Status 404 returned error can't find the container with id fba28d6fd77b7dc4e8263b3678873d36015c510fdc9df5c810adb309e8067a07
Sep 17 16:24:57.482806 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.482770    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"]
Sep 17 16:24:57.486392 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.486374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.488642 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.488616    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 16:24:57.488774 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.488708    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 16:24:57.488823 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.488776    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 16:24:57.505941 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.505914    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"]
Sep 17 16:24:57.593086 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.593048    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9444c9d7-5035-43e4-9ed8-bcba13848e7d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.593284 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.593128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrbr\" (UniqueName: \"kubernetes.io/projected/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-api-access-bwrbr\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.593284 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.593185    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.593284 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.593222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.593416 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.593281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.694089 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.694053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.694264 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.694096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.694338 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.694310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9444c9d7-5035-43e4-9ed8-bcba13848e7d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.694419 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.694401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrbr\" (UniqueName: \"kubernetes.io/projected/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-api-access-bwrbr\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.694497 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.694486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.694569 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.694497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.694705 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:24:57.694678    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 16:24:57.694811 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:24:57.694748    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9444c9d7-5035-43e4-9ed8-bcba13848e7d-proxy-tls podName:9444c9d7-5035-43e4-9ed8-bcba13848e7d nodeName:}" failed. No retries permitted until 2026-09-17 16:24:58.194729662 +0000 UTC m=+2789.118304228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9444c9d7-5035-43e4-9ed8-bcba13848e7d-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" (UID: "9444c9d7-5035-43e4-9ed8-bcba13848e7d") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Sep 17 16:24:57.694860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.694847    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.695160 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.695137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.705451 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.705422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrbr\" (UniqueName: \"kubernetes.io/projected/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-api-access-bwrbr\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:57.714931 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.714893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" event={"ID":"88c5538b-7a58-4ccc-ad85-bd72543446eb","Type":"ContainerStarted","Data":"461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7"}
Sep 17 16:24:57.715038 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:57.714965    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" event={"ID":"88c5538b-7a58-4ccc-ad85-bd72543446eb","Type":"ContainerStarted","Data":"fba28d6fd77b7dc4e8263b3678873d36015c510fdc9df5c810adb309e8067a07"}
Sep 17 16:24:58.199291 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:58.199205    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9444c9d7-5035-43e4-9ed8-bcba13848e7d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:58.201627 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:58.201604    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9444c9d7-5035-43e4-9ed8-bcba13848e7d-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:58.399010 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:58.398960    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:24:58.676245 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:58.676212    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"]
Sep 17 16:24:58.679276 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:24:58.679240    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9444c9d7_5035_43e4_9ed8_bcba13848e7d.slice/crio-d1ee8e53fedf5999de62c26bff44a4d37004a12e84e6ca8883874c816b27a8cb WatchSource:0}: Error finding container d1ee8e53fedf5999de62c26bff44a4d37004a12e84e6ca8883874c816b27a8cb: Status 404 returned error can't find the container with id d1ee8e53fedf5999de62c26bff44a4d37004a12e84e6ca8883874c816b27a8cb
Sep 17 16:24:58.721078 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:58.721048    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" event={"ID":"9444c9d7-5035-43e4-9ed8-bcba13848e7d","Type":"ContainerStarted","Data":"d1ee8e53fedf5999de62c26bff44a4d37004a12e84e6ca8883874c816b27a8cb"}
Sep 17 16:24:59.727868 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:24:59.727837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" event={"ID":"9444c9d7-5035-43e4-9ed8-bcba13848e7d","Type":"ContainerStarted","Data":"bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251"}
Sep 17 16:25:00.734508 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:00.734454    2581 generic.go:356] "Generic (PLEG): container finished" podID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerID="bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251" exitCode=0
Sep 17 16:25:00.734508 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:00.734494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" event={"ID":"9444c9d7-5035-43e4-9ed8-bcba13848e7d","Type":"ContainerDied","Data":"bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251"}
Sep 17 16:25:01.740845 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:01.740809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" event={"ID":"9444c9d7-5035-43e4-9ed8-bcba13848e7d","Type":"ContainerStarted","Data":"644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c"}
Sep 17 16:25:01.741271 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:01.740879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" event={"ID":"9444c9d7-5035-43e4-9ed8-bcba13848e7d","Type":"ContainerStarted","Data":"b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c"}
Sep 17 16:25:01.741271 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:01.740917    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:25:01.742375 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:01.742350    2581 generic.go:356] "Generic (PLEG): container finished" podID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerID="461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7" exitCode=0
Sep 17 16:25:01.742504 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:01.742409    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" event={"ID":"88c5538b-7a58-4ccc-ad85-bd72543446eb","Type":"ContainerDied","Data":"461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7"}
Sep 17 16:25:01.743628 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:01.743614    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:25:01.775561 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:01.775515    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podStartSLOduration=3.642245022 podStartE2EDuration="4.775499371s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.133254349s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:01.773806185 +0000 UTC m=+2792.697380753" watchObservedRunningTime="2026-09-17 16:25:01.775499371 +0000 UTC m=+2792.699073937"
Sep 17 16:25:02.746649 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:02.746610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:25:02.748014 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:02.747987    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:25:03.751553 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:03.751505    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:25:08.756143 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:08.756113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:25:08.756697 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:08.756665    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:25:13.804761 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:13.804728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" event={"ID":"88c5538b-7a58-4ccc-ad85-bd72543446eb","Type":"ContainerStarted","Data":"0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6"}
Sep 17 16:25:14.810992 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:14.810955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" event={"ID":"88c5538b-7a58-4ccc-ad85-bd72543446eb","Type":"ContainerStarted","Data":"439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed"}
Sep 17 16:25:14.811496 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:14.811141    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:25:14.811496 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:14.811189    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:25:14.812507 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:14.812478    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:25:14.832651 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:14.832604    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" podStartSLOduration=2.349237222 podStartE2EDuration="18.832588446s" totalImagesPullingTime="11.959394945s" totalInitContainerRuntime="4.523956279s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:24:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:25:14.831537482 +0000 UTC m=+2805.755112051" watchObservedRunningTime="2026-09-17 16:25:14.832588446 +0000 UTC m=+2805.756163016"
Sep 17 16:25:15.815726 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:15.815680    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:25:18.756596 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:18.756553    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:25:20.819829 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:20.819796    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:25:20.820537 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:20.820506    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:25:28.756739 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:28.756702    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:25:30.820950 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:30.820912    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:25:38.756950 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:38.756911    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:25:40.820894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:40.820855    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:25:48.757108 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:48.757068    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:25:50.821331 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:50.821285    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 17 16:25:58.757485 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:25:58.757426    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:26:00.821640 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:00.821610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:26:07.773359 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:07.773320    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"]
Sep 17 16:26:07.773783 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:07.773697    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container" containerID="cri-o://0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6" gracePeriod=30
Sep 17 16:26:07.773783 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:07.773726    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kube-rbac-proxy" containerID="cri-o://439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed" gracePeriod=30
Sep 17 16:26:08.046327 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:08.046296    2581 generic.go:356] "Generic (PLEG): container finished" podID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerID="439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed" exitCode=2
Sep 17 16:26:08.046513 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:08.046373    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" event={"ID":"88c5538b-7a58-4ccc-ad85-bd72543446eb","Type":"ContainerDied","Data":"439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed"}
Sep 17 16:26:08.757257 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:08.757218    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:26:10.612576 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.612548    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:26:10.750236 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.750146    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88c5538b-7a58-4ccc-ad85-bd72543446eb-proxy-tls\") pod \"88c5538b-7a58-4ccc-ad85-bd72543446eb\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") "
Sep 17 16:26:10.750236 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.750183    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88c5538b-7a58-4ccc-ad85-bd72543446eb-kserve-provision-location\") pod \"88c5538b-7a58-4ccc-ad85-bd72543446eb\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") "
Sep 17 16:26:10.750483 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.750262    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-rbac-proxy-sar-config\") pod \"88c5538b-7a58-4ccc-ad85-bd72543446eb\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") "
Sep 17 16:26:10.750483 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.750336    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j26ft\" (UniqueName: \"kubernetes.io/projected/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-api-access-j26ft\") pod \"88c5538b-7a58-4ccc-ad85-bd72543446eb\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") "
Sep 17 16:26:10.750483 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.750417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-cabundle-cert\") pod \"88c5538b-7a58-4ccc-ad85-bd72543446eb\" (UID: \"88c5538b-7a58-4ccc-ad85-bd72543446eb\") "
Sep 17 16:26:10.750658 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.750570    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88c5538b-7a58-4ccc-ad85-bd72543446eb" (UID: "88c5538b-7a58-4ccc-ad85-bd72543446eb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:26:10.750759 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.750734    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88c5538b-7a58-4ccc-ad85-bd72543446eb" (UID: "88c5538b-7a58-4ccc-ad85-bd72543446eb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:26:10.750835 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.750809    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:10.750835 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.750823    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:10.752348 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.752320    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c5538b-7a58-4ccc-ad85-bd72543446eb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88c5538b-7a58-4ccc-ad85-bd72543446eb" (UID: "88c5538b-7a58-4ccc-ad85-bd72543446eb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:26:10.752492 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.752438    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-api-access-j26ft" (OuterVolumeSpecName: "kube-api-access-j26ft") pod "88c5538b-7a58-4ccc-ad85-bd72543446eb" (UID: "88c5538b-7a58-4ccc-ad85-bd72543446eb"). InnerVolumeSpecName "kube-api-access-j26ft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:26:10.760066 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.760045    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c5538b-7a58-4ccc-ad85-bd72543446eb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88c5538b-7a58-4ccc-ad85-bd72543446eb" (UID: "88c5538b-7a58-4ccc-ad85-bd72543446eb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:26:10.851204 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.851174    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88c5538b-7a58-4ccc-ad85-bd72543446eb-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:10.851204 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.851196    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88c5538b-7a58-4ccc-ad85-bd72543446eb-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:10.851204 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:10.851205    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j26ft\" (UniqueName: \"kubernetes.io/projected/88c5538b-7a58-4ccc-ad85-bd72543446eb-kube-api-access-j26ft\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:11.061649 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.061621    2581 generic.go:356] "Generic (PLEG): container finished" podID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerID="0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6" exitCode=0
Sep 17 16:26:11.061649 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.061654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" event={"ID":"88c5538b-7a58-4ccc-ad85-bd72543446eb","Type":"ContainerDied","Data":"0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6"}
Sep 17 16:26:11.061897 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.061686    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"
Sep 17 16:26:11.061897 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.061700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5" event={"ID":"88c5538b-7a58-4ccc-ad85-bd72543446eb","Type":"ContainerDied","Data":"fba28d6fd77b7dc4e8263b3678873d36015c510fdc9df5c810adb309e8067a07"}
Sep 17 16:26:11.061897 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.061728    2581 scope.go:117] "RemoveContainer" containerID="439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed"
Sep 17 16:26:11.070250 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.070153    2581 scope.go:117] "RemoveContainer" containerID="0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6"
Sep 17 16:26:11.077589 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.077569    2581 scope.go:117] "RemoveContainer" containerID="461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7"
Sep 17 16:26:11.084474 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.084448    2581 scope.go:117] "RemoveContainer" containerID="439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed"
Sep 17 16:26:11.084763 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:26:11.084738    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed\": container with ID starting with 439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed not found: ID does not exist" containerID="439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed"
Sep 17 16:26:11.084814 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.084773    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed"} err="failed to get container status \"439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed\": rpc error: code = NotFound desc = could not find container \"439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed\": container with ID starting with 439122675d7c37b482bbd10e8103e1f3de78cc40b31b71e177c40701de7842ed not found: ID does not exist"
Sep 17 16:26:11.084814 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.084791    2581 scope.go:117] "RemoveContainer" containerID="0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6"
Sep 17 16:26:11.085068 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:26:11.085050    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6\": container with ID starting with 0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6 not found: ID does not exist" containerID="0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6"
Sep 17 16:26:11.085141 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.085073    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6"} err="failed to get container status \"0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6\": rpc error: code = NotFound desc = could not find container \"0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6\": container with ID starting with 0422601107f5e3dd92e72aee6973014f1c64b47646e434bfb50ccd82539c7bc6 not found: ID does not exist"
Sep 17 16:26:11.085141 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.085089    2581 scope.go:117] "RemoveContainer" containerID="461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7"
Sep 17 16:26:11.085340 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:26:11.085318    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7\": container with ID starting with 461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7 not found: ID does not exist" containerID="461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7"
Sep 17 16:26:11.085388 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.085347    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7"} err="failed to get container status \"461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7\": rpc error: code = NotFound desc = could not find container \"461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7\": container with ID starting with 461cda0c812c52c1aba356f3f892a0bf98a2b7654e5cc20b6cec7870ef2271a7 not found: ID does not exist"
Sep 17 16:26:11.088312 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.088294    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"]
Sep 17 16:26:11.094570 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.094552    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-pzrq5"]
Sep 17 16:26:11.557363 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:11.557333    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" path="/var/lib/kubelet/pods/88c5538b-7a58-4ccc-ad85-bd72543446eb/volumes"
Sep 17 16:26:18.757606 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:18.757574    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:26:27.464352 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:27.464316    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"]
Sep 17 16:26:27.464915 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:27.464638    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" containerID="cri-o://b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c" gracePeriod=30
Sep 17 16:26:27.464915 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:27.464668    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kube-rbac-proxy" containerID="cri-o://644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c" gracePeriod=30
Sep 17 16:26:28.133218 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.133185    2581 generic.go:356] "Generic (PLEG): container finished" podID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerID="644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c" exitCode=2
Sep 17 16:26:28.133391 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.133257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" event={"ID":"9444c9d7-5035-43e4-9ed8-bcba13848e7d","Type":"ContainerDied","Data":"644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c"}
Sep 17 16:26:28.587038 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.587011    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"]
Sep 17 16:26:28.587412 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.587319    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container"
Sep 17 16:26:28.587412 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.587330    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container"
Sep 17 16:26:28.587412 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.587345    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kube-rbac-proxy"
Sep 17 16:26:28.587412 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.587350    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kube-rbac-proxy"
Sep 17 16:26:28.587412 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.587362    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="storage-initializer"
Sep 17 16:26:28.587412 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.587369    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="storage-initializer"
Sep 17 16:26:28.587666 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.587427    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kube-rbac-proxy"
Sep 17 16:26:28.587666 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.587436    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c5538b-7a58-4ccc-ad85-bd72543446eb" containerName="kserve-container"
Sep 17 16:26:28.590629 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.590613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.592561 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.592539    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 17 16:26:28.592767 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.592749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 17 16:26:28.603949 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.603927    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"]
Sep 17 16:26:28.612378 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.612356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lpfb\" (UniqueName: \"kubernetes.io/projected/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-api-access-5lpfb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.612512 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.612415    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e772cc6e-a3cc-4835-8e61-62263e63f980-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.612512 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.612432    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.612624 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.612504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e772cc6e-a3cc-4835-8e61-62263e63f980-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.713048 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.713013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e772cc6e-a3cc-4835-8e61-62263e63f980-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.713048 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.713049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.713273 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.713070    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e772cc6e-a3cc-4835-8e61-62263e63f980-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.713273 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.713114    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lpfb\" (UniqueName: \"kubernetes.io/projected/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-api-access-5lpfb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.713539 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.713510    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e772cc6e-a3cc-4835-8e61-62263e63f980-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.713793 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.713771    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.715481 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.715449    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e772cc6e-a3cc-4835-8e61-62263e63f980-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.722325 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.722305    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lpfb\" (UniqueName: \"kubernetes.io/projected/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-api-access-5lpfb\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:28.752023 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.751988    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.55:8643/healthz\": dial tcp 10.132.0.55:8643: connect: connection refused"
Sep 17 16:26:28.757339 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.757317    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 17 16:26:28.900905 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:28.900814    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:29.174258 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:29.174176    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"]
Sep 17 16:26:29.177215 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:26:29.177186    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode772cc6e_a3cc_4835_8e61_62263e63f980.slice/crio-c65d0bcdd3ac9432be689910b2d95e5280f382a84e90bd3f8a611f8ae7ec1ef0 WatchSource:0}: Error finding container c65d0bcdd3ac9432be689910b2d95e5280f382a84e90bd3f8a611f8ae7ec1ef0: Status 404 returned error can't find the container with id c65d0bcdd3ac9432be689910b2d95e5280f382a84e90bd3f8a611f8ae7ec1ef0
Sep 17 16:26:30.143343 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:30.143310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng" event={"ID":"e772cc6e-a3cc-4835-8e61-62263e63f980","Type":"ContainerStarted","Data":"48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5"}
Sep 17 16:26:30.143725 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:30.143391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng" event={"ID":"e772cc6e-a3cc-4835-8e61-62263e63f980","Type":"ContainerStarted","Data":"c65d0bcdd3ac9432be689910b2d95e5280f382a84e90bd3f8a611f8ae7ec1ef0"}
Sep 17 16:26:32.102431 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.102409    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:26:32.136438 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.136409    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9444c9d7-5035-43e4-9ed8-bcba13848e7d-proxy-tls\") pod \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") "
Sep 17 16:26:32.136600 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.136459    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-rbac-proxy-sar-config\") pod \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") "
Sep 17 16:26:32.136600 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.136502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kserve-provision-location\") pod \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") "
Sep 17 16:26:32.136600 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.136567    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-cabundle-cert\") pod \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") "
Sep 17 16:26:32.136600 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.136598    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwrbr\" (UniqueName: \"kubernetes.io/projected/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-api-access-bwrbr\") pod \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\" (UID: \"9444c9d7-5035-43e4-9ed8-bcba13848e7d\") "
Sep 17 16:26:32.136895 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.136866    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9444c9d7-5035-43e4-9ed8-bcba13848e7d" (UID: "9444c9d7-5035-43e4-9ed8-bcba13848e7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:26:32.136960 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.136906    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9444c9d7-5035-43e4-9ed8-bcba13848e7d" (UID: "9444c9d7-5035-43e4-9ed8-bcba13848e7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:26:32.136960 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.136920    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9444c9d7-5035-43e4-9ed8-bcba13848e7d" (UID: "9444c9d7-5035-43e4-9ed8-bcba13848e7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:26:32.138583 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.138555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-api-access-bwrbr" (OuterVolumeSpecName: "kube-api-access-bwrbr") pod "9444c9d7-5035-43e4-9ed8-bcba13848e7d" (UID: "9444c9d7-5035-43e4-9ed8-bcba13848e7d"). InnerVolumeSpecName "kube-api-access-bwrbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:26:32.138583 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.138565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9444c9d7-5035-43e4-9ed8-bcba13848e7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9444c9d7-5035-43e4-9ed8-bcba13848e7d" (UID: "9444c9d7-5035-43e4-9ed8-bcba13848e7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:26:32.153115 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.153091    2581 generic.go:356] "Generic (PLEG): container finished" podID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerID="b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c" exitCode=0
Sep 17 16:26:32.153221 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.153152    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"
Sep 17 16:26:32.153221 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.153167    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" event={"ID":"9444c9d7-5035-43e4-9ed8-bcba13848e7d","Type":"ContainerDied","Data":"b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c"}
Sep 17 16:26:32.153344 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.153228    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8" event={"ID":"9444c9d7-5035-43e4-9ed8-bcba13848e7d","Type":"ContainerDied","Data":"d1ee8e53fedf5999de62c26bff44a4d37004a12e84e6ca8883874c816b27a8cb"}
Sep 17 16:26:32.153344 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.153262    2581 scope.go:117] "RemoveContainer" containerID="644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c"
Sep 17 16:26:32.161924 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.161907    2581 scope.go:117] "RemoveContainer" containerID="b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c"
Sep 17 16:26:32.169330 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.169313    2581 scope.go:117] "RemoveContainer" containerID="bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251"
Sep 17 16:26:32.176634 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.176617    2581 scope.go:117] "RemoveContainer" containerID="644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c"
Sep 17 16:26:32.176910 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:26:32.176889    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c\": container with ID starting with 644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c not found: ID does not exist" containerID="644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c"
Sep 17 16:26:32.176958 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.176919    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c"} err="failed to get container status \"644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c\": rpc error: code = NotFound desc = could not find container \"644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c\": container with ID starting with 644a88468164def8c1b3ac791ff58a8b0ac527382dcaad2c0efb1a18348dfa1c not found: ID does not exist"
Sep 17 16:26:32.176958 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.176938    2581 scope.go:117] "RemoveContainer" containerID="b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c"
Sep 17 16:26:32.177161 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:26:32.177144    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c\": container with ID starting with b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c not found: ID does not exist" containerID="b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c"
Sep 17 16:26:32.177202 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.177167    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c"} err="failed to get container status \"b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c\": rpc error: code = NotFound desc = could not find container \"b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c\": container with ID starting with b0d95f1f99d7252179335b6e1ebbdcff2476cfa500b00db02102300b2b0b6e3c not found: ID does not exist"
Sep 17 16:26:32.177202 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.177182    2581 scope.go:117] "RemoveContainer" containerID="bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251"
Sep 17 16:26:32.177389 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:26:32.177373    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251\": container with ID starting with bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251 not found: ID does not exist" containerID="bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251"
Sep 17 16:26:32.177426 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.177393    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251"} err="failed to get container status \"bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251\": rpc error: code = NotFound desc = could not find container \"bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251\": container with ID starting with bd7ed7c8669c71483d967b9c07e55bce1ece80babff181693822ed07127ad251 not found: ID does not exist"
Sep 17 16:26:32.185651 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.185626    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"]
Sep 17 16:26:32.197182 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.197158    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676f88c84-6z9r8"]
Sep 17 16:26:32.237104 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.237070    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9444c9d7-5035-43e4-9ed8-bcba13848e7d-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:32.237104 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.237101    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:32.237244 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.237112    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:32.237244 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.237121    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9444c9d7-5035-43e4-9ed8-bcba13848e7d-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:32.237244 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:32.237130    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwrbr\" (UniqueName: \"kubernetes.io/projected/9444c9d7-5035-43e4-9ed8-bcba13848e7d-kube-api-access-bwrbr\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:33.558127 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:33.558095    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" path="/var/lib/kubelet/pods/9444c9d7-5035-43e4-9ed8-bcba13848e7d/volumes"
Sep 17 16:26:35.168536 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:35.168436    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng_e772cc6e-a3cc-4835-8e61-62263e63f980/storage-initializer/0.log"
Sep 17 16:26:35.168536 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:35.168490    2581 generic.go:356] "Generic (PLEG): container finished" podID="e772cc6e-a3cc-4835-8e61-62263e63f980" containerID="48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5" exitCode=1
Sep 17 16:26:35.168929 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:35.168568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng" event={"ID":"e772cc6e-a3cc-4835-8e61-62263e63f980","Type":"ContainerDied","Data":"48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5"}
Sep 17 16:26:36.174525 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:36.174497    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng_e772cc6e-a3cc-4835-8e61-62263e63f980/storage-initializer/0.log"
Sep 17 16:26:36.174907 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:36.174543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng" event={"ID":"e772cc6e-a3cc-4835-8e61-62263e63f980","Type":"ContainerStarted","Data":"9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3"}
Sep 17 16:26:38.607825 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:38.607793    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"]
Sep 17 16:26:38.608218 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:38.608015    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng" podUID="e772cc6e-a3cc-4835-8e61-62263e63f980" containerName="storage-initializer" containerID="cri-o://9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3" gracePeriod=30
Sep 17 16:26:40.649791 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.649768    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng_e772cc6e-a3cc-4835-8e61-62263e63f980/storage-initializer/1.log"
Sep 17 16:26:40.650212 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.650196    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng_e772cc6e-a3cc-4835-8e61-62263e63f980/storage-initializer/0.log"
Sep 17 16:26:40.653037 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.653017    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:40.703070 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.703030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e772cc6e-a3cc-4835-8e61-62263e63f980-kserve-provision-location\") pod \"e772cc6e-a3cc-4835-8e61-62263e63f980\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") "
Sep 17 16:26:40.703070 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.703086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-rbac-proxy-sar-config\") pod \"e772cc6e-a3cc-4835-8e61-62263e63f980\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") "
Sep 17 16:26:40.703351 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.703113    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lpfb\" (UniqueName: \"kubernetes.io/projected/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-api-access-5lpfb\") pod \"e772cc6e-a3cc-4835-8e61-62263e63f980\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") "
Sep 17 16:26:40.703351 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.703178    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e772cc6e-a3cc-4835-8e61-62263e63f980-proxy-tls\") pod \"e772cc6e-a3cc-4835-8e61-62263e63f980\" (UID: \"e772cc6e-a3cc-4835-8e61-62263e63f980\") "
Sep 17 16:26:40.703494 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.703370    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e772cc6e-a3cc-4835-8e61-62263e63f980-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e772cc6e-a3cc-4835-8e61-62263e63f980" (UID: "e772cc6e-a3cc-4835-8e61-62263e63f980"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:26:40.703603 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.703577    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e772cc6e-a3cc-4835-8e61-62263e63f980" (UID: "e772cc6e-a3cc-4835-8e61-62263e63f980"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:26:40.705280 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.705255    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e772cc6e-a3cc-4835-8e61-62263e63f980-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e772cc6e-a3cc-4835-8e61-62263e63f980" (UID: "e772cc6e-a3cc-4835-8e61-62263e63f980"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:26:40.705400 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.705378    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-api-access-5lpfb" (OuterVolumeSpecName: "kube-api-access-5lpfb") pod "e772cc6e-a3cc-4835-8e61-62263e63f980" (UID: "e772cc6e-a3cc-4835-8e61-62263e63f980"). InnerVolumeSpecName "kube-api-access-5lpfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:26:40.804574 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.804539    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e772cc6e-a3cc-4835-8e61-62263e63f980-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:40.804574 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.804571    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:40.804775 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.804581    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lpfb\" (UniqueName: \"kubernetes.io/projected/e772cc6e-a3cc-4835-8e61-62263e63f980-kube-api-access-5lpfb\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:40.804775 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:40.804594    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e772cc6e-a3cc-4835-8e61-62263e63f980-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:26:41.196692 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.196608    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng_e772cc6e-a3cc-4835-8e61-62263e63f980/storage-initializer/1.log"
Sep 17 16:26:41.197005 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.196988    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng_e772cc6e-a3cc-4835-8e61-62263e63f980/storage-initializer/0.log"
Sep 17 16:26:41.197099 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.197028    2581 generic.go:356] "Generic (PLEG): container finished" podID="e772cc6e-a3cc-4835-8e61-62263e63f980" containerID="9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3" exitCode=1
Sep 17 16:26:41.197099 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.197078    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"
Sep 17 16:26:41.197181 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.197111    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng" event={"ID":"e772cc6e-a3cc-4835-8e61-62263e63f980","Type":"ContainerDied","Data":"9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3"}
Sep 17 16:26:41.197217 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.197194    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng" event={"ID":"e772cc6e-a3cc-4835-8e61-62263e63f980","Type":"ContainerDied","Data":"c65d0bcdd3ac9432be689910b2d95e5280f382a84e90bd3f8a611f8ae7ec1ef0"}
Sep 17 16:26:41.197250 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.197222    2581 scope.go:117] "RemoveContainer" containerID="9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3"
Sep 17 16:26:41.205935 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.205918    2581 scope.go:117] "RemoveContainer" containerID="48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5"
Sep 17 16:26:41.213177 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.213159    2581 scope.go:117] "RemoveContainer" containerID="9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3"
Sep 17 16:26:41.213478 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:26:41.213445    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3\": container with ID starting with 9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3 not found: ID does not exist" containerID="9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3"
Sep 17 16:26:41.213522 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.213493    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3"} err="failed to get container status \"9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3\": rpc error: code = NotFound desc = could not find container \"9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3\": container with ID starting with 9b9d1b8ebab1c14c7010a297c4e618ffd72de295a191125cef41a1e2c0e7b3a3 not found: ID does not exist"
Sep 17 16:26:41.213522 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.213510    2581 scope.go:117] "RemoveContainer" containerID="48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5"
Sep 17 16:26:41.213770 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:26:41.213753    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5\": container with ID starting with 48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5 not found: ID does not exist" containerID="48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5"
Sep 17 16:26:41.213810 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.213777    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5"} err="failed to get container status \"48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5\": rpc error: code = NotFound desc = could not find container \"48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5\": container with ID starting with 48d803f890366435508ccb4a36b9d5e32a7fec81d426516b2b822d18565b79b5 not found: ID does not exist"
Sep 17 16:26:41.241305 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.241276    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"]
Sep 17 16:26:41.251907 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.251874    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-cb799f68-vthng"]
Sep 17 16:26:41.557328 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:26:41.557294    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e772cc6e-a3cc-4835-8e61-62263e63f980" path="/var/lib/kubelet/pods/e772cc6e-a3cc-4835-8e61-62263e63f980/volumes"
Sep 17 16:27:29.642835 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.642797    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"]
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643107    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kube-rbac-proxy"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643117    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kube-rbac-proxy"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643124    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e772cc6e-a3cc-4835-8e61-62263e63f980" containerName="storage-initializer"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643130    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e772cc6e-a3cc-4835-8e61-62263e63f980" containerName="storage-initializer"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643148    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643154    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643165    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="storage-initializer"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643170    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="storage-initializer"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643213    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e772cc6e-a3cc-4835-8e61-62263e63f980" containerName="storage-initializer"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643222    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kube-rbac-proxy"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643229    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9444c9d7-5035-43e4-9ed8-bcba13848e7d" containerName="kserve-container"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643269    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e772cc6e-a3cc-4835-8e61-62263e63f980" containerName="storage-initializer"
Sep 17 16:27:29.643315 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643274    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e772cc6e-a3cc-4835-8e61-62263e63f980" containerName="storage-initializer"
Sep 17 16:27:29.643753 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.643332    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e772cc6e-a3cc-4835-8e61-62263e63f980" containerName="storage-initializer"
Sep 17 16:27:29.646569 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.646550    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.648924 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.648894    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:27:29.649070 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.648925    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:27:29.649247 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.649227    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:27:29.649629 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.649614    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 17 16:27:29.649700 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.649635    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:27:29.649920 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.649904    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 16:27:29.650118 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.650106    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:27:29.657621 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.657598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"]
Sep 17 16:27:29.805588 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.805545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f16476-8d20-43ec-89db-a35823966895-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.805588 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.805589    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkf2b\" (UniqueName: \"kubernetes.io/projected/d6f16476-8d20-43ec-89db-a35823966895-kube-api-access-xkf2b\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.805794 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.805707    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f16476-8d20-43ec-89db-a35823966895-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.805794 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.805743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.805870 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.805793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.906958 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.906866    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkf2b\" (UniqueName: \"kubernetes.io/projected/d6f16476-8d20-43ec-89db-a35823966895-kube-api-access-xkf2b\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.906958 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.906933    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f16476-8d20-43ec-89db-a35823966895-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.907151 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.906967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.907151 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.907009    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.907151 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.907050    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f16476-8d20-43ec-89db-a35823966895-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.907392 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.907368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f16476-8d20-43ec-89db-a35823966895-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.907686 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.907649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.907686 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.907675    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.909507 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.909491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f16476-8d20-43ec-89db-a35823966895-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.916855 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.916837    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkf2b\" (UniqueName: \"kubernetes.io/projected/d6f16476-8d20-43ec-89db-a35823966895-kube-api-access-xkf2b\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:29.957388 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:29.957349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:30.235325 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:30.235301    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"]
Sep 17 16:27:30.237891 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:27:30.237857    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f16476_8d20_43ec_89db_a35823966895.slice/crio-187d64179fb0fafacb2f2af50879bfd1e430b94fcfe9fac35200eed64deec890 WatchSource:0}: Error finding container 187d64179fb0fafacb2f2af50879bfd1e430b94fcfe9fac35200eed64deec890: Status 404 returned error can't find the container with id 187d64179fb0fafacb2f2af50879bfd1e430b94fcfe9fac35200eed64deec890
Sep 17 16:27:30.401164 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:30.401121    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" event={"ID":"d6f16476-8d20-43ec-89db-a35823966895","Type":"ContainerStarted","Data":"ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036"}
Sep 17 16:27:30.401333 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:30.401196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" event={"ID":"d6f16476-8d20-43ec-89db-a35823966895","Type":"ContainerStarted","Data":"187d64179fb0fafacb2f2af50879bfd1e430b94fcfe9fac35200eed64deec890"}
Sep 17 16:27:35.421029 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:35.420947    2581 generic.go:356] "Generic (PLEG): container finished" podID="d6f16476-8d20-43ec-89db-a35823966895" containerID="ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036" exitCode=0
Sep 17 16:27:35.421029 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:35.421019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" event={"ID":"d6f16476-8d20-43ec-89db-a35823966895","Type":"ContainerDied","Data":"ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036"}
Sep 17 16:27:36.426836 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:36.426801    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" event={"ID":"d6f16476-8d20-43ec-89db-a35823966895","Type":"ContainerStarted","Data":"2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7"}
Sep 17 16:27:36.427282 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:36.426861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" event={"ID":"d6f16476-8d20-43ec-89db-a35823966895","Type":"ContainerStarted","Data":"39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3"}
Sep 17 16:27:36.427282 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:36.427192    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:36.427390 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:36.427334    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:36.428695 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:36.428670    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 16:27:36.447100 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:36.447052    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podStartSLOduration=2.69749147 podStartE2EDuration="7.44703894s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.74954747s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:27:36.446107554 +0000 UTC m=+2947.369682122" watchObservedRunningTime="2026-09-17 16:27:36.44703894 +0000 UTC m=+2947.370613507"
Sep 17 16:27:37.431055 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:37.431018    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 16:27:42.436457 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:42.436417    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:27:42.436922 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:42.436895    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 16:27:52.437360 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:27:52.437276    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 16:28:02.437547 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:02.437503    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 16:28:10.898779 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:10.898744    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"]
Sep 17 16:28:10.903657 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:10.903634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:10.905618 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:10.905597    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 17 16:28:10.905955 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:10.905940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 16:28:10.906305 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:10.906288    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 17 16:28:10.917937 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:10.917917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"]
Sep 17 16:28:11.055535 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.055501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.055710 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.055549    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.055710 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.055570    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.055710 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.055618    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29pdw\" (UniqueName: \"kubernetes.io/projected/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-api-access-29pdw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.156357 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.156266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.156357 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.156308    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.156357 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.156326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.156683 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.156517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29pdw\" (UniqueName: \"kubernetes.io/projected/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-api-access-29pdw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.156773 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.156753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.157119 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.157102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.158862 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.158844    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.164350 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.164332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29pdw\" (UniqueName: \"kubernetes.io/projected/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-api-access-29pdw\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.213294 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.213259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:11.493192 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.493165    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"]
Sep 17 16:28:11.495295 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:28:11.495267    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eded92a_3b13_4110_b2bb_c6e3ebf9ea72.slice/crio-a643a0ff05c9ddffd8522cd0959cb93df7256db691cfcd0de4243b2e68d514ec WatchSource:0}: Error finding container a643a0ff05c9ddffd8522cd0959cb93df7256db691cfcd0de4243b2e68d514ec: Status 404 returned error can't find the container with id a643a0ff05c9ddffd8522cd0959cb93df7256db691cfcd0de4243b2e68d514ec
Sep 17 16:28:11.574272 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.574237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr" event={"ID":"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72","Type":"ContainerStarted","Data":"d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653"}
Sep 17 16:28:11.574450 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:11.574295    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr" event={"ID":"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72","Type":"ContainerStarted","Data":"a643a0ff05c9ddffd8522cd0959cb93df7256db691cfcd0de4243b2e68d514ec"}
Sep 17 16:28:12.437438 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:12.437401    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 16:28:16.595200 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:16.595174    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr_1eded92a-3b13-4110-b2bb-c6e3ebf9ea72/storage-initializer/0.log"
Sep 17 16:28:16.595558 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:16.595211    2581 generic.go:356] "Generic (PLEG): container finished" podID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" containerID="d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653" exitCode=1
Sep 17 16:28:16.595558 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:16.595284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr" event={"ID":"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72","Type":"ContainerDied","Data":"d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653"}
Sep 17 16:28:17.600903 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:17.600873    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr_1eded92a-3b13-4110-b2bb-c6e3ebf9ea72/storage-initializer/0.log"
Sep 17 16:28:17.601290 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:17.600990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr" event={"ID":"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72","Type":"ContainerStarted","Data":"41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2"}
Sep 17 16:28:20.845682 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:20.845647    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"]
Sep 17 16:28:20.846178 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:20.845926    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr" podUID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" containerName="storage-initializer" containerID="cri-o://41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2" gracePeriod=30
Sep 17 16:28:22.437816 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:22.437788    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:28:24.181897 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.181872    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr_1eded92a-3b13-4110-b2bb-c6e3ebf9ea72/storage-initializer/1.log"
Sep 17 16:28:24.182269 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.182255    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr_1eded92a-3b13-4110-b2bb-c6e3ebf9ea72/storage-initializer/0.log"
Sep 17 16:28:24.184615 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.184600    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:24.271456 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.271364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29pdw\" (UniqueName: \"kubernetes.io/projected/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-api-access-29pdw\") pod \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") "
Sep 17 16:28:24.271456 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.271409    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-proxy-tls\") pod \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") "
Sep 17 16:28:24.271456 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.271437    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-rbac-proxy-sar-config\") pod \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") "
Sep 17 16:28:24.271722 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.271483    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kserve-provision-location\") pod \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\" (UID: \"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72\") "
Sep 17 16:28:24.271828 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.271793    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" (UID: "1eded92a-3b13-4110-b2bb-c6e3ebf9ea72"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:24.271828 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.271799    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" (UID: "1eded92a-3b13-4110-b2bb-c6e3ebf9ea72"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:28:24.273454 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.273423    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-api-access-29pdw" (OuterVolumeSpecName: "kube-api-access-29pdw") pod "1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" (UID: "1eded92a-3b13-4110-b2bb-c6e3ebf9ea72"). InnerVolumeSpecName "kube-api-access-29pdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:24.273558 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.273490    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" (UID: "1eded92a-3b13-4110-b2bb-c6e3ebf9ea72"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:24.372989 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.372945    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29pdw\" (UniqueName: \"kubernetes.io/projected/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-api-access-29pdw\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:28:24.372989 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.372979    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:28:24.372989 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.372989    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:28:24.372989 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.372998    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:28:24.632292 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.632266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr_1eded92a-3b13-4110-b2bb-c6e3ebf9ea72/storage-initializer/1.log"
Sep 17 16:28:24.632721 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.632706    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr_1eded92a-3b13-4110-b2bb-c6e3ebf9ea72/storage-initializer/0.log"
Sep 17 16:28:24.632820 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.632739    2581 generic.go:356] "Generic (PLEG): container finished" podID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" containerID="41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2" exitCode=1
Sep 17 16:28:24.632820 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.632770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr" event={"ID":"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72","Type":"ContainerDied","Data":"41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2"}
Sep 17 16:28:24.632942 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.632815    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"
Sep 17 16:28:24.632942 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.632832    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr" event={"ID":"1eded92a-3b13-4110-b2bb-c6e3ebf9ea72","Type":"ContainerDied","Data":"a643a0ff05c9ddffd8522cd0959cb93df7256db691cfcd0de4243b2e68d514ec"}
Sep 17 16:28:24.632942 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.632860    2581 scope.go:117] "RemoveContainer" containerID="41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2"
Sep 17 16:28:24.641497 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.641457    2581 scope.go:117] "RemoveContainer" containerID="d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653"
Sep 17 16:28:24.648804 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.648783    2581 scope.go:117] "RemoveContainer" containerID="41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2"
Sep 17 16:28:24.649065 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:28:24.649046    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2\": container with ID starting with 41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2 not found: ID does not exist" containerID="41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2"
Sep 17 16:28:24.649138 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.649078    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2"} err="failed to get container status \"41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2\": rpc error: code = NotFound desc = could not find container \"41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2\": container with ID starting with 41bd719af7d8780e721dd813bd5f4145209b1d6c4a98eb8e0705b13f8273bcf2 not found: ID does not exist"
Sep 17 16:28:24.649138 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.649105    2581 scope.go:117] "RemoveContainer" containerID="d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653"
Sep 17 16:28:24.649337 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:28:24.649311    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653\": container with ID starting with d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653 not found: ID does not exist" containerID="d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653"
Sep 17 16:28:24.649379 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.649342    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653"} err="failed to get container status \"d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653\": rpc error: code = NotFound desc = could not find container \"d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653\": container with ID starting with d8d7bc080afc301d0e4f35672bc49fbd76e482d2ba82cf946755025bcfce5653 not found: ID does not exist"
Sep 17 16:28:24.671865 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.671836    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"]
Sep 17 16:28:24.674751 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:24.674725    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-7fcc99b96d-59dnr"]
Sep 17 16:28:25.561688 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:25.561654    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" path="/var/lib/kubelet/pods/1eded92a-3b13-4110-b2bb-c6e3ebf9ea72/volumes"
Sep 17 16:28:31.251514 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:31.251481    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"]
Sep 17 16:28:31.251916 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:31.251707    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container" containerID="cri-o://39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3" gracePeriod=30
Sep 17 16:28:31.251916 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:31.251778    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kube-rbac-proxy" containerID="cri-o://2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7" gracePeriod=30
Sep 17 16:28:31.663394 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:31.663360    2581 generic.go:356] "Generic (PLEG): container finished" podID="d6f16476-8d20-43ec-89db-a35823966895" containerID="2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7" exitCode=2
Sep 17 16:28:31.663571 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:31.663432    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" event={"ID":"d6f16476-8d20-43ec-89db-a35823966895","Type":"ContainerDied","Data":"2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7"}
Sep 17 16:28:32.431390 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:32.431331    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.57:8643/healthz\": dial tcp 10.132.0.57:8643: connect: connection refused"
Sep 17 16:28:32.436891 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:32.436864    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.57:8080: connect: connection refused"
Sep 17 16:28:34.085703 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.085675    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:28:34.157913 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.157884    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-kube-rbac-proxy-sar-config\") pod \"d6f16476-8d20-43ec-89db-a35823966895\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") "
Sep 17 16:28:34.158081 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.157936    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f16476-8d20-43ec-89db-a35823966895-kserve-provision-location\") pod \"d6f16476-8d20-43ec-89db-a35823966895\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") "
Sep 17 16:28:34.158081 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.157962    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-cabundle-cert\") pod \"d6f16476-8d20-43ec-89db-a35823966895\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") "
Sep 17 16:28:34.158081 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.158026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkf2b\" (UniqueName: \"kubernetes.io/projected/d6f16476-8d20-43ec-89db-a35823966895-kube-api-access-xkf2b\") pod \"d6f16476-8d20-43ec-89db-a35823966895\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") "
Sep 17 16:28:34.158081 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.158064    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f16476-8d20-43ec-89db-a35823966895-proxy-tls\") pod \"d6f16476-8d20-43ec-89db-a35823966895\" (UID: \"d6f16476-8d20-43ec-89db-a35823966895\") "
Sep 17 16:28:34.158358 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.158328    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6f16476-8d20-43ec-89db-a35823966895" (UID: "d6f16476-8d20-43ec-89db-a35823966895"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:34.158428 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.158347    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d6f16476-8d20-43ec-89db-a35823966895" (UID: "d6f16476-8d20-43ec-89db-a35823966895"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:28:34.160178 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.160143    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f16476-8d20-43ec-89db-a35823966895-kube-api-access-xkf2b" (OuterVolumeSpecName: "kube-api-access-xkf2b") pod "d6f16476-8d20-43ec-89db-a35823966895" (UID: "d6f16476-8d20-43ec-89db-a35823966895"). InnerVolumeSpecName "kube-api-access-xkf2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:28:34.160281 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.160253    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6f16476-8d20-43ec-89db-a35823966895-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6f16476-8d20-43ec-89db-a35823966895" (UID: "d6f16476-8d20-43ec-89db-a35823966895"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:28:34.167895 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.167874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6f16476-8d20-43ec-89db-a35823966895-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d6f16476-8d20-43ec-89db-a35823966895" (UID: "d6f16476-8d20-43ec-89db-a35823966895"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:28:34.258929 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.258838    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:28:34.258929 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.258878    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d6f16476-8d20-43ec-89db-a35823966895-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:28:34.258929 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.258888    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d6f16476-8d20-43ec-89db-a35823966895-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:28:34.258929 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.258898    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkf2b\" (UniqueName: \"kubernetes.io/projected/d6f16476-8d20-43ec-89db-a35823966895-kube-api-access-xkf2b\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:28:34.258929 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.258906    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6f16476-8d20-43ec-89db-a35823966895-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:28:34.677238 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.677204    2581 generic.go:356] "Generic (PLEG): container finished" podID="d6f16476-8d20-43ec-89db-a35823966895" containerID="39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3" exitCode=0
Sep 17 16:28:34.677414 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.677253    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"
Sep 17 16:28:34.677414 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.677284    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" event={"ID":"d6f16476-8d20-43ec-89db-a35823966895","Type":"ContainerDied","Data":"39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3"}
Sep 17 16:28:34.677414 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.677344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4" event={"ID":"d6f16476-8d20-43ec-89db-a35823966895","Type":"ContainerDied","Data":"187d64179fb0fafacb2f2af50879bfd1e430b94fcfe9fac35200eed64deec890"}
Sep 17 16:28:34.677414 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.677372    2581 scope.go:117] "RemoveContainer" containerID="2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7"
Sep 17 16:28:34.685920 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.685904    2581 scope.go:117] "RemoveContainer" containerID="39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3"
Sep 17 16:28:34.693168 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.693153    2581 scope.go:117] "RemoveContainer" containerID="ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036"
Sep 17 16:28:34.699518 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.699499    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"]
Sep 17 16:28:34.700095 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.700077    2581 scope.go:117] "RemoveContainer" containerID="2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7"
Sep 17 16:28:34.700359 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:28:34.700341    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7\": container with ID starting with 2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7 not found: ID does not exist" containerID="2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7"
Sep 17 16:28:34.700414 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.700366    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7"} err="failed to get container status \"2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7\": rpc error: code = NotFound desc = could not find container \"2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7\": container with ID starting with 2415c7e7cbf094511adff6043c4f76a4686e222ba6260b02cc9ff884179152e7 not found: ID does not exist"
Sep 17 16:28:34.700414 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.700381    2581 scope.go:117] "RemoveContainer" containerID="39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3"
Sep 17 16:28:34.700658 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:28:34.700638    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3\": container with ID starting with 39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3 not found: ID does not exist" containerID="39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3"
Sep 17 16:28:34.700709 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.700661    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3"} err="failed to get container status \"39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3\": rpc error: code = NotFound desc = could not find container \"39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3\": container with ID starting with 39e6d1a1cd18ae4693d25bdea3370658c4d71601b57688e0a495ac5864d64af3 not found: ID does not exist"
Sep 17 16:28:34.700709 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.700674    2581 scope.go:117] "RemoveContainer" containerID="ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036"
Sep 17 16:28:34.700887 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:28:34.700872    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036\": container with ID starting with ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036 not found: ID does not exist" containerID="ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036"
Sep 17 16:28:34.700927 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.700890    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036"} err="failed to get container status \"ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036\": rpc error: code = NotFound desc = could not find container \"ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036\": container with ID starting with ea61a08a24ad538257e4c73b00e2b176cc70188ebc4be56dcff261dc7daa1036 not found: ID does not exist"
Sep 17 16:28:34.705229 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:34.705209    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wvjk4"]
Sep 17 16:28:35.557335 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:35.557302    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6f16476-8d20-43ec-89db-a35823966895" path="/var/lib/kubelet/pods/d6f16476-8d20-43ec-89db-a35823966895/volumes"
Sep 17 16:28:36.487284 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487251    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"]
Sep 17 16:28:36.487585 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487573    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="storage-initializer"
Sep 17 16:28:36.487643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487587    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="storage-initializer"
Sep 17 16:28:36.487643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487602    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container"
Sep 17 16:28:36.487643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487608    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container"
Sep 17 16:28:36.487643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487618    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" containerName="storage-initializer"
Sep 17 16:28:36.487643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487623    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" containerName="storage-initializer"
Sep 17 16:28:36.487643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487632    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" containerName="storage-initializer"
Sep 17 16:28:36.487643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487637    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" containerName="storage-initializer"
Sep 17 16:28:36.487643 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487645    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kube-rbac-proxy"
Sep 17 16:28:36.487888 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487652    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kube-rbac-proxy"
Sep 17 16:28:36.487888 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487696    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" containerName="storage-initializer"
Sep 17 16:28:36.487888 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487703    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eded92a-3b13-4110-b2bb-c6e3ebf9ea72" containerName="storage-initializer"
Sep 17 16:28:36.487888 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487710    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kube-rbac-proxy"
Sep 17 16:28:36.487888 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.487715    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6f16476-8d20-43ec-89db-a35823966895" containerName="kserve-container"
Sep 17 16:28:36.492409 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.492391    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.494340 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.494315    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:28:36.494487 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.494317    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:28:36.494637 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.494621    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:28:36.494721 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.494662    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 17 16:28:36.494721 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.494689    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:28:36.494721 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.494701    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:28:36.494897 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.494691    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 17 16:28:36.500261 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.500242    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"]
Sep 17 16:28:36.578263 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.578227    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.578683 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.578267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.578683 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.578349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.578683 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.578395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.578683 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.578428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8t9k\" (UniqueName: \"kubernetes.io/projected/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-api-access-t8t9k\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.679615 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.679588    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8t9k\" (UniqueName: \"kubernetes.io/projected/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-api-access-t8t9k\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.679787 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.679626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.679787 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.679651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.679787 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.679689    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.679787 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.679717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.679787 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:28:36.679775    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Sep 17 16:28:36.680063 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:28:36.679836    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-proxy-tls podName:ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1 nodeName:}" failed. No retries permitted until 2026-09-17 16:28:37.179818169 +0000 UTC m=+3008.103392733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-proxy-tls") pod "isvc-pmml-predictor-8674795f59-tq9pm" (UID: "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1") : secret "isvc-pmml-predictor-serving-cert" not found
Sep 17 16:28:36.680135 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.680115    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.680335 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.680317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.680404 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.680351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:36.693388 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:36.693365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8t9k\" (UniqueName: \"kubernetes.io/projected/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-api-access-t8t9k\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:37.183833 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:37.183797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:37.186111 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:37.186082    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-tq9pm\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:37.403167 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:37.403131    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:37.675568 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:37.675544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"]
Sep 17 16:28:37.677782 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:28:37.677754    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef786c8d_cb14_4bc0_86fb_5aae48ddb4b1.slice/crio-a39f7482471a375d3b7e1bce6ae45b680676909070bbadb7f398f355e4e96a3a WatchSource:0}: Error finding container a39f7482471a375d3b7e1bce6ae45b680676909070bbadb7f398f355e4e96a3a: Status 404 returned error can't find the container with id a39f7482471a375d3b7e1bce6ae45b680676909070bbadb7f398f355e4e96a3a
Sep 17 16:28:37.693503 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:37.693453    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" event={"ID":"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1","Type":"ContainerStarted","Data":"a39f7482471a375d3b7e1bce6ae45b680676909070bbadb7f398f355e4e96a3a"}
Sep 17 16:28:38.699349 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:38.699314    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" event={"ID":"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1","Type":"ContainerStarted","Data":"52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5"}
Sep 17 16:28:41.713801 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:41.713772    2581 generic.go:356] "Generic (PLEG): container finished" podID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerID="52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5" exitCode=0
Sep 17 16:28:41.714128 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:41.713817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" event={"ID":"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1","Type":"ContainerDied","Data":"52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5"}
Sep 17 16:28:49.759885 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:49.759838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" event={"ID":"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1","Type":"ContainerStarted","Data":"e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955"}
Sep 17 16:28:49.760307 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:49.759909    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" event={"ID":"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1","Type":"ContainerStarted","Data":"a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19"}
Sep 17 16:28:49.760307 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:49.760095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:49.782108 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:49.782067    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podStartSLOduration=2.371727573 podStartE2EDuration="13.782054987s" totalImagesPullingTime="7.460728449s" totalInitContainerRuntime="3.949598965s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:28:49.780845658 +0000 UTC m=+3020.704420235" watchObservedRunningTime="2026-09-17 16:28:49.782054987 +0000 UTC m=+3020.705629553"
Sep 17 16:28:50.764145 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:50.764118    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:50.765390 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:50.765359    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:28:51.768842 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:51.768804    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:28:56.773067 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:56.773041    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:28:56.773590 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:28:56.773563    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:29:06.774303 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:29:06.774266    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:29:16.773899 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:29:16.773859    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:29:26.773893 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:29:26.773796    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:29:36.774050 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:29:36.774012    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:29:46.774272 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:29:46.774234    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:29:56.773566 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:29:56.773533    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:30:06.773955 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:06.773917    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 17 16:30:16.774506 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:16.774478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:30:28.451958 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:28.451924    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"]
Sep 17 16:30:28.452548 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:28.452154    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container" containerID="cri-o://a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19" gracePeriod=30
Sep 17 16:30:28.452548 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:28.452204    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kube-rbac-proxy" containerID="cri-o://e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955" gracePeriod=30
Sep 17 16:30:29.171239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:29.171203    2581 generic.go:356] "Generic (PLEG): container finished" podID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerID="e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955" exitCode=2
Sep 17 16:30:29.171413 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:29.171248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" event={"ID":"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1","Type":"ContainerDied","Data":"e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955"}
Sep 17 16:30:31.769376 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:31.769337    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.59:8643/healthz\": dial tcp 10.132.0.59:8643: connect: connection refused"
Sep 17 16:30:32.596323 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.596300    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:30:32.661289 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.661253    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-cabundle-cert\") pod \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") "
Sep 17 16:30:32.661289 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.661301    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-rbac-proxy-sar-config\") pod \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") "
Sep 17 16:30:32.661555 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.661384    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8t9k\" (UniqueName: \"kubernetes.io/projected/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-api-access-t8t9k\") pod \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") "
Sep 17 16:30:32.661864 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.661842    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-proxy-tls\") pod \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") "
Sep 17 16:30:32.661981 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.661928    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kserve-provision-location\") pod \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\" (UID: \"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1\") "
Sep 17 16:30:32.662232 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.662204    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" (UID: "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:30:32.662404 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.662209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" (UID: "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:30:32.662532 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.662302    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" (UID: "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:30:32.662629 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.662608    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.662729 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.662716    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.663959 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.663934    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" (UID: "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:30:32.664358 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.664343    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-api-access-t8t9k" (OuterVolumeSpecName: "kube-api-access-t8t9k") pod "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" (UID: "ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1"). InnerVolumeSpecName "kube-api-access-t8t9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:30:32.764191 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.764106    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.764191 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.764140    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8t9k\" (UniqueName: \"kubernetes.io/projected/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-kube-api-access-t8t9k\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:30:32.764191 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:32.764152    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:30:33.190814 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.190782    2581 generic.go:356] "Generic (PLEG): container finished" podID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerID="a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19" exitCode=0
Sep 17 16:30:33.191208 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.190851    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"
Sep 17 16:30:33.191208 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.190876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" event={"ID":"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1","Type":"ContainerDied","Data":"a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19"}
Sep 17 16:30:33.191208 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.190962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm" event={"ID":"ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1","Type":"ContainerDied","Data":"a39f7482471a375d3b7e1bce6ae45b680676909070bbadb7f398f355e4e96a3a"}
Sep 17 16:30:33.191208 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.190992    2581 scope.go:117] "RemoveContainer" containerID="e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955"
Sep 17 16:30:33.199753 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.199738    2581 scope.go:117] "RemoveContainer" containerID="a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19"
Sep 17 16:30:33.207103 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.207087    2581 scope.go:117] "RemoveContainer" containerID="52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5"
Sep 17 16:30:33.212597 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.212576    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"]
Sep 17 16:30:33.214606 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.214591    2581 scope.go:117] "RemoveContainer" containerID="e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955"
Sep 17 16:30:33.214869 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:30:33.214853    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955\": container with ID starting with e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955 not found: ID does not exist" containerID="e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955"
Sep 17 16:30:33.214933 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.214888    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955"} err="failed to get container status \"e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955\": rpc error: code = NotFound desc = could not find container \"e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955\": container with ID starting with e8d033c3d868212dde08c36d9afe9abf979f3d627d7d5a20dbd227709adf1955 not found: ID does not exist"
Sep 17 16:30:33.214933 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.214905    2581 scope.go:117] "RemoveContainer" containerID="a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19"
Sep 17 16:30:33.215148 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:30:33.215131    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19\": container with ID starting with a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19 not found: ID does not exist" containerID="a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19"
Sep 17 16:30:33.215210 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.215153    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19"} err="failed to get container status \"a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19\": rpc error: code = NotFound desc = could not find container \"a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19\": container with ID starting with a0c1718fed56e9031f9fa89adb76b7ed8e6be04a8c08a9f0910e8222ce543e19 not found: ID does not exist"
Sep 17 16:30:33.215210 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.215168    2581 scope.go:117] "RemoveContainer" containerID="52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5"
Sep 17 16:30:33.215417 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:30:33.215403    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5\": container with ID starting with 52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5 not found: ID does not exist" containerID="52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5"
Sep 17 16:30:33.215474 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.215420    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5"} err="failed to get container status \"52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5\": rpc error: code = NotFound desc = could not find container \"52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5\": container with ID starting with 52ffa23bd41ade1621e261ba241b58f802ba3e01b23b35afe370cf75ec3dc8a5 not found: ID does not exist"
Sep 17 16:30:33.218668 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.218642    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-tq9pm"]
Sep 17 16:30:33.557547 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.557507    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" path="/var/lib/kubelet/pods/ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1/volumes"
Sep 17 16:30:33.639978 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.639942    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"]
Sep 17 16:30:33.640267 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.640256    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kube-rbac-proxy"
Sep 17 16:30:33.640313 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.640269    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kube-rbac-proxy"
Sep 17 16:30:33.640313 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.640280    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container"
Sep 17 16:30:33.640313 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.640285    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container"
Sep 17 16:30:33.640313 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.640301    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="storage-initializer"
Sep 17 16:30:33.640313 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.640306    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="storage-initializer"
Sep 17 16:30:33.640491 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.640352    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kserve-container"
Sep 17 16:30:33.640491 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.640360    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef786c8d-cb14-4bc0-86fb-5aae48ddb4b1" containerName="kube-rbac-proxy"
Sep 17 16:30:33.644890 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.644872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.647053 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.647022    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Sep 17 16:30:33.647196 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.647065    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 16:30:33.647196 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.647100    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:30:33.647196 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.647069    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:30:33.647316 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.647069    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:30:33.647542 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.647521    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-rrzd7\""
Sep 17 16:30:33.647667 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.647582    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Sep 17 16:30:33.652007 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.651984    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"]
Sep 17 16:30:33.772225 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.772189    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.772402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.772237    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.772402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.772324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba67804d-b2b1-472a-9837-1e4e740cfd5a-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.772402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.772351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.772402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.772374    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tncm\" (UniqueName: \"kubernetes.io/projected/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-api-access-2tncm\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.873640 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.873542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba67804d-b2b1-472a-9837-1e4e740cfd5a-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.873640 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.873584    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.873640 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.873605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tncm\" (UniqueName: \"kubernetes.io/projected/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-api-access-2tncm\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.873640 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.873634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.873956 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.873663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.873956 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:30:33.873792    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-runtime-predictor-serving-cert: secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 17 16:30:33.873956 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:30:33.873884    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba67804d-b2b1-472a-9837-1e4e740cfd5a-proxy-tls podName:ba67804d-b2b1-472a-9837-1e4e740cfd5a nodeName:}" failed. No retries permitted until 2026-09-17 16:30:34.373861927 +0000 UTC m=+3125.297436472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ba67804d-b2b1-472a-9837-1e4e740cfd5a-proxy-tls") pod "isvc-pmml-runtime-predictor-885fc8777-txjc6" (UID: "ba67804d-b2b1-472a-9837-1e4e740cfd5a") : secret "isvc-pmml-runtime-predictor-serving-cert" not found
Sep 17 16:30:33.874182 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.874166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.874289 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.874273    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.874370 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.874354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:33.882645 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:33.882620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tncm\" (UniqueName: \"kubernetes.io/projected/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-api-access-2tncm\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:34.378914 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:34.378876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba67804d-b2b1-472a-9837-1e4e740cfd5a-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:34.381350 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:34.381326    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba67804d-b2b1-472a-9837-1e4e740cfd5a-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-txjc6\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:34.555969 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:34.555910    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:34.833079 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:34.833042    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"]
Sep 17 16:30:34.836597 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:30:34.836570    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba67804d_b2b1_472a_9837_1e4e740cfd5a.slice/crio-8dba2d1e526d7b98bc5e72378a0a060c5f46559629100610f271bb15285e77d9 WatchSource:0}: Error finding container 8dba2d1e526d7b98bc5e72378a0a060c5f46559629100610f271bb15285e77d9: Status 404 returned error can't find the container with id 8dba2d1e526d7b98bc5e72378a0a060c5f46559629100610f271bb15285e77d9
Sep 17 16:30:35.202800 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:35.202699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" event={"ID":"ba67804d-b2b1-472a-9837-1e4e740cfd5a","Type":"ContainerStarted","Data":"a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3"}
Sep 17 16:30:35.202800 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:35.202756    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" event={"ID":"ba67804d-b2b1-472a-9837-1e4e740cfd5a","Type":"ContainerStarted","Data":"8dba2d1e526d7b98bc5e72378a0a060c5f46559629100610f271bb15285e77d9"}
Sep 17 16:30:39.221972 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:39.221934    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerID="a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3" exitCode=0
Sep 17 16:30:39.222451 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:39.222011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" event={"ID":"ba67804d-b2b1-472a-9837-1e4e740cfd5a","Type":"ContainerDied","Data":"a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3"}
Sep 17 16:30:40.228761 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:40.228726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" event={"ID":"ba67804d-b2b1-472a-9837-1e4e740cfd5a","Type":"ContainerStarted","Data":"895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633"}
Sep 17 16:30:40.229147 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:40.228814    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" event={"ID":"ba67804d-b2b1-472a-9837-1e4e740cfd5a","Type":"ContainerStarted","Data":"44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9"}
Sep 17 16:30:40.229147 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:40.229095    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:40.229272 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:40.229251    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:40.230528 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:40.230499    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:30:40.253520 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:40.253456    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podStartSLOduration=3.462259351 podStartE2EDuration="7.25344272s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.791183369s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:30:40.252561514 +0000 UTC m=+3131.176136082" watchObservedRunningTime="2026-09-17 16:30:40.25344272 +0000 UTC m=+3131.177017287"
Sep 17 16:30:41.233588 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:41.233548    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:30:46.238187 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:46.238161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:30:46.238689 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:46.238661    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:30:56.238836 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:30:56.238743    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:31:06.238655 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:31:06.238605    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:31:16.238660 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:31:16.238622    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:31:26.239559 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:31:26.239515    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:31:36.239614 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:31:36.239572    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:31:46.238603 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:31:46.238562    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:31:50.552693 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:31:50.552654    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:32:00.553434 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:00.553387    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 17 16:32:10.553635 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:10.553600    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:32:15.485052 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:15.485018    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"]
Sep 17 16:32:15.485422 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:15.485317    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container" containerID="cri-o://44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9" gracePeriod=30
Sep 17 16:32:15.485422 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:15.485378    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kube-rbac-proxy" containerID="cri-o://895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633" gracePeriod=30
Sep 17 16:32:15.621772 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:15.621737    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerID="895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633" exitCode=2
Sep 17 16:32:15.621913 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:15.621795    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" event={"ID":"ba67804d-b2b1-472a-9837-1e4e740cfd5a","Type":"ContainerDied","Data":"895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633"}
Sep 17 16:32:16.234598 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:16.234556    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Sep 17 16:32:19.628823 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.628800    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:32:19.649488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.649441    2581 generic.go:356] "Generic (PLEG): container finished" podID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerID="44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9" exitCode=0
Sep 17 16:32:19.649617 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.649512    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" event={"ID":"ba67804d-b2b1-472a-9837-1e4e740cfd5a","Type":"ContainerDied","Data":"44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9"}
Sep 17 16:32:19.649617 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.649533    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"
Sep 17 16:32:19.649617 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.649589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6" event={"ID":"ba67804d-b2b1-472a-9837-1e4e740cfd5a","Type":"ContainerDied","Data":"8dba2d1e526d7b98bc5e72378a0a060c5f46559629100610f271bb15285e77d9"}
Sep 17 16:32:19.649770 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.649628    2581 scope.go:117] "RemoveContainer" containerID="895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633"
Sep 17 16:32:19.658724 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.658702    2581 scope.go:117] "RemoveContainer" containerID="44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9"
Sep 17 16:32:19.665910 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.665893    2581 scope.go:117] "RemoveContainer" containerID="a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3"
Sep 17 16:32:19.673228 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.673212    2581 scope.go:117] "RemoveContainer" containerID="895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633"
Sep 17 16:32:19.673500 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:32:19.673480    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633\": container with ID starting with 895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633 not found: ID does not exist" containerID="895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633"
Sep 17 16:32:19.673563 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.673509    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633"} err="failed to get container status \"895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633\": rpc error: code = NotFound desc = could not find container \"895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633\": container with ID starting with 895d1230859209ad49b26958a3e1afd0d7af10471e4f1001989a9e57aabde633 not found: ID does not exist"
Sep 17 16:32:19.673563 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.673527    2581 scope.go:117] "RemoveContainer" containerID="44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9"
Sep 17 16:32:19.673802 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:32:19.673784    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9\": container with ID starting with 44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9 not found: ID does not exist" containerID="44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9"
Sep 17 16:32:19.673854 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.673807    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9"} err="failed to get container status \"44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9\": rpc error: code = NotFound desc = could not find container \"44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9\": container with ID starting with 44592685c80b5b804454732f0bc8befc48390f63b46aa929967ddb82bc16c1b9 not found: ID does not exist"
Sep 17 16:32:19.673854 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.673824    2581 scope.go:117] "RemoveContainer" containerID="a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3"
Sep 17 16:32:19.674064 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:32:19.674049    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3\": container with ID starting with a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3 not found: ID does not exist" containerID="a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3"
Sep 17 16:32:19.674105 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.674068    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3"} err="failed to get container status \"a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3\": rpc error: code = NotFound desc = could not find container \"a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3\": container with ID starting with a7941a1296b119d5e4c4b5f0e340fada0add4221ac5eaf0b80f5aad3f50b5de3 not found: ID does not exist"
Sep 17 16:32:19.757538 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.757516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba67804d-b2b1-472a-9837-1e4e740cfd5a-proxy-tls\") pod \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") "
Sep 17 16:32:19.757638 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.757549    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tncm\" (UniqueName: \"kubernetes.io/projected/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-api-access-2tncm\") pod \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") "
Sep 17 16:32:19.757638 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.757603    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-cabundle-cert\") pod \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") "
Sep 17 16:32:19.757727 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.757635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-rbac-proxy-sar-config\") pod \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") "
Sep 17 16:32:19.757727 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.757679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kserve-provision-location\") pod \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\" (UID: \"ba67804d-b2b1-472a-9837-1e4e740cfd5a\") "
Sep 17 16:32:19.758001 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.757979    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ba67804d-b2b1-472a-9837-1e4e740cfd5a" (UID: "ba67804d-b2b1-472a-9837-1e4e740cfd5a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:32:19.758001 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.757989    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ba67804d-b2b1-472a-9837-1e4e740cfd5a" (UID: "ba67804d-b2b1-472a-9837-1e4e740cfd5a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:32:19.758110 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.758015    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ba67804d-b2b1-472a-9837-1e4e740cfd5a" (UID: "ba67804d-b2b1-472a-9837-1e4e740cfd5a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:32:19.759666 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.759638    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba67804d-b2b1-472a-9837-1e4e740cfd5a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ba67804d-b2b1-472a-9837-1e4e740cfd5a" (UID: "ba67804d-b2b1-472a-9837-1e4e740cfd5a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:32:19.759769 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.759748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-api-access-2tncm" (OuterVolumeSpecName: "kube-api-access-2tncm") pod "ba67804d-b2b1-472a-9837-1e4e740cfd5a" (UID: "ba67804d-b2b1-472a-9837-1e4e740cfd5a"). InnerVolumeSpecName "kube-api-access-2tncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:32:19.858918 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.858891    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:32:19.858918 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.858915    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:32:19.858918 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.858924    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ba67804d-b2b1-472a-9837-1e4e740cfd5a-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:32:19.859100 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.858932    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tncm\" (UniqueName: \"kubernetes.io/projected/ba67804d-b2b1-472a-9837-1e4e740cfd5a-kube-api-access-2tncm\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:32:19.859100 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.858941    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ba67804d-b2b1-472a-9837-1e4e740cfd5a-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:32:19.971119 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.971090    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"]
Sep 17 16:32:19.975132 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:19.975103    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-txjc6"]
Sep 17 16:32:21.556972 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:32:21.556939    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" path="/var/lib/kubelet/pods/ba67804d-b2b1-472a-9837-1e4e740cfd5a/volumes"
Sep 17 16:34:05.937569 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.937530    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"]
Sep 17 16:34:05.938103 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.938008    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="storage-initializer"
Sep 17 16:34:05.938103 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.938027    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="storage-initializer"
Sep 17 16:34:05.938103 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.938040    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kube-rbac-proxy"
Sep 17 16:34:05.938103 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.938048    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kube-rbac-proxy"
Sep 17 16:34:05.938103 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.938062    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container"
Sep 17 16:34:05.938103 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.938070    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container"
Sep 17 16:34:05.938510 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.938133    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kserve-container"
Sep 17 16:34:05.938510 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.938152    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba67804d-b2b1-472a-9837-1e4e740cfd5a" containerName="kube-rbac-proxy"
Sep 17 16:34:05.941653 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.941632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:05.943751 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.943730    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 16:34:05.943902 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.943841    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 16:34:05.943902 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.943856    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-859eae\""
Sep 17 16:34:05.943902 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.943884    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-859eae-dockercfg-lkswr\""
Sep 17 16:34:05.944116 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.943944    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-859eae-predictor-serving-cert\""
Sep 17 16:34:05.944116 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.943890    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-859eae-kube-rbac-proxy-sar-config\""
Sep 17 16:34:05.944215 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.944194    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 16:34:05.950650 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:05.950629    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"]
Sep 17 16:34:06.048155 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.048115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.048155 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.048154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kserve-provision-location\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.048375 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.048172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-proxy-tls\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.048375 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.048266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbwvs\" (UniqueName: \"kubernetes.io/projected/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-api-access-wbwvs\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.048375 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.048300    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-cabundle-cert\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.148734 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.148704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbwvs\" (UniqueName: \"kubernetes.io/projected/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-api-access-wbwvs\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.148894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.148738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-cabundle-cert\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.148894 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.148775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.149005 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.148960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kserve-provision-location\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.149060 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.149008    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-proxy-tls\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.149160 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:34:06.149143    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-859eae-predictor-serving-cert: secret "isvc-secondary-859eae-predictor-serving-cert" not found
Sep 17 16:34:06.149222 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:34:06.149212    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-proxy-tls podName:cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5 nodeName:}" failed. No retries permitted until 2026-09-17 16:34:06.649196725 +0000 UTC m=+3337.572771274 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-proxy-tls") pod "isvc-secondary-859eae-predictor-54d5c46d5f-blvk6" (UID: "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5") : secret "isvc-secondary-859eae-predictor-serving-cert" not found
Sep 17 16:34:06.149331 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.149309    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kserve-provision-location\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.149426 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.149410    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.149426 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.149417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-cabundle-cert\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.157105 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.157081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbwvs\" (UniqueName: \"kubernetes.io/projected/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-api-access-wbwvs\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.653241 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.653199    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-proxy-tls\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.655577 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.655558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-proxy-tls\") pod \"isvc-secondary-859eae-predictor-54d5c46d5f-blvk6\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") " pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:06.852959 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:06.852914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:07.126408 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:07.126374    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"]
Sep 17 16:34:07.128927 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:34:07.128899    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc5f08d_c348_4cfd_bb6a_e2941a00ccf5.slice/crio-83ecb0ff42140e25cfae5898d1b13e2f5357799abf5bf418cbd15c39d271cde3 WatchSource:0}: Error finding container 83ecb0ff42140e25cfae5898d1b13e2f5357799abf5bf418cbd15c39d271cde3: Status 404 returned error can't find the container with id 83ecb0ff42140e25cfae5898d1b13e2f5357799abf5bf418cbd15c39d271cde3
Sep 17 16:34:08.079697 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:08.079663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6" event={"ID":"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5","Type":"ContainerStarted","Data":"cdc8f2e7cca3ba86c1e57e7ef25cb3d6f342470d81f730b33540a6ba07beed97"}
Sep 17 16:34:08.079867 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:08.079721    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6" event={"ID":"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5","Type":"ContainerStarted","Data":"83ecb0ff42140e25cfae5898d1b13e2f5357799abf5bf418cbd15c39d271cde3"}
Sep 17 16:34:13.101833 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:13.101807    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5/storage-initializer/0.log"
Sep 17 16:34:13.102199 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:13.101848    2581 generic.go:356] "Generic (PLEG): container finished" podID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" containerID="cdc8f2e7cca3ba86c1e57e7ef25cb3d6f342470d81f730b33540a6ba07beed97" exitCode=1
Sep 17 16:34:13.102199 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:13.101928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6" event={"ID":"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5","Type":"ContainerDied","Data":"cdc8f2e7cca3ba86c1e57e7ef25cb3d6f342470d81f730b33540a6ba07beed97"}
Sep 17 16:34:14.107859 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:14.107831    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5/storage-initializer/0.log"
Sep 17 16:34:14.108242 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:14.107949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6" event={"ID":"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5","Type":"ContainerStarted","Data":"4c0dcd308728e90e77c99896e2d4cf53a5f553df7a28f3ad2f045ad94f0cdff2"}
Sep 17 16:34:20.132650 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:20.132622    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5/storage-initializer/1.log"
Sep 17 16:34:20.133073 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:20.133056    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5/storage-initializer/0.log"
Sep 17 16:34:20.133120 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:20.133090    2581 generic.go:356] "Generic (PLEG): container finished" podID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" containerID="4c0dcd308728e90e77c99896e2d4cf53a5f553df7a28f3ad2f045ad94f0cdff2" exitCode=1
Sep 17 16:34:20.133187 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:20.133165    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6" event={"ID":"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5","Type":"ContainerDied","Data":"4c0dcd308728e90e77c99896e2d4cf53a5f553df7a28f3ad2f045ad94f0cdff2"}
Sep 17 16:34:20.133245 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:20.133235    2581 scope.go:117] "RemoveContainer" containerID="cdc8f2e7cca3ba86c1e57e7ef25cb3d6f342470d81f730b33540a6ba07beed97"
Sep 17 16:34:20.133591 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:20.133567    2581 scope.go:117] "RemoveContainer" containerID="cdc8f2e7cca3ba86c1e57e7ef25cb3d6f342470d81f730b33540a6ba07beed97"
Sep 17 16:34:20.143928 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:34:20.143899    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_kserve-ci-e2e-test(cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5)\"" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6" podUID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5"
Sep 17 16:34:20.144061 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:34:20.144030    2581 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_kserve-ci-e2e-test_cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5_0 in pod sandbox 83ecb0ff42140e25cfae5898d1b13e2f5357799abf5bf418cbd15c39d271cde3 from index: no such id: 'cdc8f2e7cca3ba86c1e57e7ef25cb3d6f342470d81f730b33540a6ba07beed97'" containerID="cdc8f2e7cca3ba86c1e57e7ef25cb3d6f342470d81f730b33540a6ba07beed97"
Sep 17 16:34:20.144117 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:20.144073    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc8f2e7cca3ba86c1e57e7ef25cb3d6f342470d81f730b33540a6ba07beed97"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_kserve-ci-e2e-test_cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5_0 in pod sandbox 83ecb0ff42140e25cfae5898d1b13e2f5357799abf5bf418cbd15c39d271cde3 from index: no such id: 'cdc8f2e7cca3ba86c1e57e7ef25cb3d6f342470d81f730b33540a6ba07beed97'"
Sep 17 16:34:21.138975 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:21.138948    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5/storage-initializer/1.log"
Sep 17 16:34:23.926405 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:23.926359    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"]
Sep 17 16:34:24.052165 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.052143    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5/storage-initializer/1.log"
Sep 17 16:34:24.055926 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.055905    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:24.096810 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.096777    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-rbac-proxy-sar-config\") pod \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") "
Sep 17 16:34:24.096976 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.096830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-cabundle-cert\") pod \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") "
Sep 17 16:34:24.096976 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.096884    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-proxy-tls\") pod \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") "
Sep 17 16:34:24.097098 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.096985    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kserve-provision-location\") pod \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") "
Sep 17 16:34:24.097098 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.097072    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbwvs\" (UniqueName: \"kubernetes.io/projected/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-api-access-wbwvs\") pod \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\" (UID: \"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5\") "
Sep 17 16:34:24.097252 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.097232    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" (UID: "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:24.097320 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.097254    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" (UID: "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 16:34:24.097320 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.097257    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" (UID: "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:34:24.097422 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.097361    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kserve-provision-location\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:34:24.097422 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.097380    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:34:24.097422 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.097395    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-cabundle-cert\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:34:24.098934 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.098910    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" (UID: "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 16:34:24.099036 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.098975    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-api-access-wbwvs" (OuterVolumeSpecName: "kube-api-access-wbwvs") pod "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" (UID: "cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5"). InnerVolumeSpecName "kube-api-access-wbwvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:34:24.152409 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.152381    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-859eae-predictor-54d5c46d5f-blvk6_cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5/storage-initializer/1.log"
Sep 17 16:34:24.152595 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.152477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6" event={"ID":"cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5","Type":"ContainerDied","Data":"83ecb0ff42140e25cfae5898d1b13e2f5357799abf5bf418cbd15c39d271cde3"}
Sep 17 16:34:24.152595 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.152500    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"
Sep 17 16:34:24.152595 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.152538    2581 scope.go:117] "RemoveContainer" containerID="4c0dcd308728e90e77c99896e2d4cf53a5f553df7a28f3ad2f045ad94f0cdff2"
Sep 17 16:34:24.189837 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.189763    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"]
Sep 17 16:34:24.196421 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.196392    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-859eae-predictor-54d5c46d5f-blvk6"]
Sep 17 16:34:24.198239 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.198219    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-proxy-tls\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:34:24.198323 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:24.198243    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbwvs\" (UniqueName: \"kubernetes.io/projected/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5-kube-api-access-wbwvs\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:34:25.556976 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:25.556945    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" path="/var/lib/kubelet/pods/cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5/volumes"
Sep 17 16:34:30.337750 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.337717    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mtpvm/must-gather-s6flz"]
Sep 17 16:34:30.338144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.338028    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" containerName="storage-initializer"
Sep 17 16:34:30.338144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.338039    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" containerName="storage-initializer"
Sep 17 16:34:30.338144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.338051    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" containerName="storage-initializer"
Sep 17 16:34:30.338144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.338057    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" containerName="storage-initializer"
Sep 17 16:34:30.338144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.338110    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" containerName="storage-initializer"
Sep 17 16:34:30.338144 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.338117    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfc5f08d-c348-4cfd-bb6a-e2941a00ccf5" containerName="storage-initializer"
Sep 17 16:34:30.342361 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.342342    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:34:30.344244 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.344224    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtpvm\"/\"kube-root-ca.crt\""
Sep 17 16:34:30.344430 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.344412    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mtpvm\"/\"openshift-service-ca.crt\""
Sep 17 16:34:30.349806 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.349480    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtpvm/must-gather-s6flz"]
Sep 17 16:34:30.449774 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.449739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwzv\" (UniqueName: \"kubernetes.io/projected/f6df12d7-8db1-418e-8c3e-2f3c5434f779-kube-api-access-kxwzv\") pod \"must-gather-s6flz\" (UID: \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\") " pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:34:30.449961 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.449819    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f6df12d7-8db1-418e-8c3e-2f3c5434f779-must-gather-output\") pod \"must-gather-s6flz\" (UID: \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\") " pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:34:30.550660 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.550625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f6df12d7-8db1-418e-8c3e-2f3c5434f779-must-gather-output\") pod \"must-gather-s6flz\" (UID: \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\") " pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:34:30.550855 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.550686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwzv\" (UniqueName: \"kubernetes.io/projected/f6df12d7-8db1-418e-8c3e-2f3c5434f779-kube-api-access-kxwzv\") pod \"must-gather-s6flz\" (UID: \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\") " pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:34:30.550978 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.550957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f6df12d7-8db1-418e-8c3e-2f3c5434f779-must-gather-output\") pod \"must-gather-s6flz\" (UID: \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\") " pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:34:30.559177 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.559145    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwzv\" (UniqueName: \"kubernetes.io/projected/f6df12d7-8db1-418e-8c3e-2f3c5434f779-kube-api-access-kxwzv\") pod \"must-gather-s6flz\" (UID: \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\") " pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:34:30.667617 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.667533    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:34:30.938151 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.938066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mtpvm/must-gather-s6flz"]
Sep 17 16:34:30.941268 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:34:30.941243    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6df12d7_8db1_418e_8c3e_2f3c5434f779.slice/crio-970a784e8ff0b30e6e660b70dacaedbdedfda0070c3b9ff936269dbf93b8591b WatchSource:0}: Error finding container 970a784e8ff0b30e6e660b70dacaedbdedfda0070c3b9ff936269dbf93b8591b: Status 404 returned error can't find the container with id 970a784e8ff0b30e6e660b70dacaedbdedfda0070c3b9ff936269dbf93b8591b
Sep 17 16:34:30.943542 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:30.943523    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 16:34:31.182671 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:31.182633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtpvm/must-gather-s6flz" event={"ID":"f6df12d7-8db1-418e-8c3e-2f3c5434f779","Type":"ContainerStarted","Data":"970a784e8ff0b30e6e660b70dacaedbdedfda0070c3b9ff936269dbf93b8591b"}
Sep 17 16:34:37.215819 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:37.215789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtpvm/must-gather-s6flz" event={"ID":"f6df12d7-8db1-418e-8c3e-2f3c5434f779","Type":"ContainerStarted","Data":"78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967"}
Sep 17 16:34:38.223003 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:38.222955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtpvm/must-gather-s6flz" event={"ID":"f6df12d7-8db1-418e-8c3e-2f3c5434f779","Type":"ContainerStarted","Data":"c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c"}
Sep 17 16:34:38.238193 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:34:38.238136    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mtpvm/must-gather-s6flz" podStartSLOduration=2.086294415 podStartE2EDuration="8.238119867s" totalImagesPullingTime="6.151825452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 16:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:34:38.237635881 +0000 UTC m=+3369.161210450" watchObservedRunningTime="2026-09-17 16:34:38.238119867 +0000 UTC m=+3369.161694434"
Sep 17 16:35:06.349117 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:06.349075    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" containerID="78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967" exitCode=0
Sep 17 16:35:06.349554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:06.349146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mtpvm/must-gather-s6flz" event={"ID":"f6df12d7-8db1-418e-8c3e-2f3c5434f779","Type":"ContainerDied","Data":"78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967"}
Sep 17 16:35:06.349554 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:06.349455    2581 scope.go:117] "RemoveContainer" containerID="78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967"
Sep 17 16:35:07.189453 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:07.189428    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mtpvm_must-gather-s6flz_f6df12d7-8db1-418e-8c3e-2f3c5434f779/gather/0.log"
Sep 17 16:35:11.679357 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:11.679320    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8lfn8_0f9ba41d-7cfb-423d-acbb-8fee6d495fe7/global-pull-secret-syncer/0.log"
Sep 17 16:35:11.934145 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:11.934063    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-x2wr6_83930156-498b-4193-8569-cbc39a4658f9/konnectivity-agent/0.log"
Sep 17 16:35:12.056888 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:12.056849    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-157.ec2.internal_5227f32ecce9c191f5f111ce617fad14/haproxy/0.log"
Sep 17 16:35:12.772699 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:12.772667    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-mtpvm/must-gather-s6flz"]
Sep 17 16:35:12.773110 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:12.772850    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-mtpvm/must-gather-s6flz" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" containerName="copy" containerID="cri-o://c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c" gracePeriod=2
Sep 17 16:35:12.777448 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:12.777422    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-mtpvm/must-gather-s6flz"]
Sep 17 16:35:13.002356 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.002334    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mtpvm_must-gather-s6flz_f6df12d7-8db1-418e-8c3e-2f3c5434f779/copy/0.log"
Sep 17 16:35:13.005040 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.005023    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:35:13.006822 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:35:13.006798    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-s6flz\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mtpvm\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:35:13.039202 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.039171    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxwzv\" (UniqueName: \"kubernetes.io/projected/f6df12d7-8db1-418e-8c3e-2f3c5434f779-kube-api-access-kxwzv\") pod \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\" (UID: \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\") "
Sep 17 16:35:13.039330 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.039237    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f6df12d7-8db1-418e-8c3e-2f3c5434f779-must-gather-output\") pod \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\" (UID: \"f6df12d7-8db1-418e-8c3e-2f3c5434f779\") "
Sep 17 16:35:13.040805 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.040778    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6df12d7-8db1-418e-8c3e-2f3c5434f779-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f6df12d7-8db1-418e-8c3e-2f3c5434f779" (UID: "f6df12d7-8db1-418e-8c3e-2f3c5434f779"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 16:35:13.041458 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.041439    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6df12d7-8db1-418e-8c3e-2f3c5434f779-kube-api-access-kxwzv" (OuterVolumeSpecName: "kube-api-access-kxwzv") pod "f6df12d7-8db1-418e-8c3e-2f3c5434f779" (UID: "f6df12d7-8db1-418e-8c3e-2f3c5434f779"). InnerVolumeSpecName "kube-api-access-kxwzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 16:35:13.140164 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.140128    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxwzv\" (UniqueName: \"kubernetes.io/projected/f6df12d7-8db1-418e-8c3e-2f3c5434f779-kube-api-access-kxwzv\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:35:13.140164 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.140158    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f6df12d7-8db1-418e-8c3e-2f3c5434f779-must-gather-output\") on node \"ip-10-0-140-157.ec2.internal\" DevicePath \"\""
Sep 17 16:35:13.380634 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.380557    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-mtpvm_must-gather-s6flz_f6df12d7-8db1-418e-8c3e-2f3c5434f779/copy/0.log"
Sep 17 16:35:13.380960 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.380937    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" containerID="c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c" exitCode=143
Sep 17 16:35:13.381021 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.380984    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:35:13.381064 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.381033    2581 scope.go:117] "RemoveContainer" containerID="c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c"
Sep 17 16:35:13.383556 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:35:13.383515    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-s6flz\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mtpvm\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:35:13.390591 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.390572    2581 scope.go:117] "RemoveContainer" containerID="78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967"
Sep 17 16:35:13.392062 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:35:13.392023    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-s6flz\" is forbidden: User \"system:node:ip-10-0-140-157.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-mtpvm\": no relationship found between node 'ip-10-0-140-157.ec2.internal' and this object" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" pod="openshift-must-gather-mtpvm/must-gather-s6flz"
Sep 17 16:35:13.403270 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.403246    2581 scope.go:117] "RemoveContainer" containerID="c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c"
Sep 17 16:35:13.403630 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:35:13.403604    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c\": container with ID starting with c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c not found: ID does not exist" containerID="c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c"
Sep 17 16:35:13.403733 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.403640    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c"} err="failed to get container status \"c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c\": rpc error: code = NotFound desc = could not find container \"c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c\": container with ID starting with c0bbcd17a6ebb0baca4043ab91b9de0e17ffe789ab651956c6ef3b2a3457a14c not found: ID does not exist"
Sep 17 16:35:13.403733 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.403659    2581 scope.go:117] "RemoveContainer" containerID="78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967"
Sep 17 16:35:13.403938 ip-10-0-140-157 kubenswrapper[2581]: E0917 16:35:13.403910    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967\": container with ID starting with 78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967 not found: ID does not exist" containerID="78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967"
Sep 17 16:35:13.404003 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.403941    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967"} err="failed to get container status \"78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967\": rpc error: code = NotFound desc = could not find container \"78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967\": container with ID starting with 78210583b507ee441fc1d714b407cc10f138df2334e23a3e8b24e0c3671fc967 not found: ID does not exist"
Sep 17 16:35:13.557970 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:13.557936    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" path="/var/lib/kubelet/pods/f6df12d7-8db1-418e-8c3e-2f3c5434f779/volumes"
Sep 17 16:35:16.095346 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.095317    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65313996-18e8-4205-b69a-2daa1d9cf68b/alertmanager/0.log"
Sep 17 16:35:16.126355 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.126328    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65313996-18e8-4205-b69a-2daa1d9cf68b/config-reloader/0.log"
Sep 17 16:35:16.158650 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.158620    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65313996-18e8-4205-b69a-2daa1d9cf68b/kube-rbac-proxy-web/0.log"
Sep 17 16:35:16.188550 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.188513    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65313996-18e8-4205-b69a-2daa1d9cf68b/kube-rbac-proxy/0.log"
Sep 17 16:35:16.219417 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.219385    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65313996-18e8-4205-b69a-2daa1d9cf68b/kube-rbac-proxy-metric/0.log"
Sep 17 16:35:16.252911 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.252825    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65313996-18e8-4205-b69a-2daa1d9cf68b/prom-label-proxy/0.log"
Sep 17 16:35:16.283478 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.283444    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_65313996-18e8-4205-b69a-2daa1d9cf68b/init-config-reloader/0.log"
Sep 17 16:35:16.489027 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.489001    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-fs6mh_ee283caa-b59d-4122-a21d-80ef64e92fb7/monitoring-plugin/0.log"
Sep 17 16:35:16.531385 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.531298    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8bwq_4646ca52-be14-4602-b081-160d471768ec/node-exporter/0.log"
Sep 17 16:35:16.564522 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.564493    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8bwq_4646ca52-be14-4602-b081-160d471768ec/kube-rbac-proxy/0.log"
Sep 17 16:35:16.593650 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.593625    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s8bwq_4646ca52-be14-4602-b081-160d471768ec/init-textfile/0.log"
Sep 17 16:35:16.833851 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.833811    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-9bp6w_7e095828-1ae8-4785-be45-76814c0d11fb/kube-rbac-proxy-main/0.log"
Sep 17 16:35:16.859613 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.859581    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-9bp6w_7e095828-1ae8-4785-be45-76814c0d11fb/kube-rbac-proxy-self/0.log"
Sep 17 16:35:16.889004 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.888979    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-9bp6w_7e095828-1ae8-4785-be45-76814c0d11fb/openshift-state-metrics/0.log"
Sep 17 16:35:16.940243 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.940211    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe81b2e4-16f1-4338-b968-c9fcc694c207/prometheus/0.log"
Sep 17 16:35:16.967566 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:16.967530    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe81b2e4-16f1-4338-b968-c9fcc694c207/config-reloader/0.log"
Sep 17 16:35:17.002909 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.002884    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe81b2e4-16f1-4338-b968-c9fcc694c207/thanos-sidecar/0.log"
Sep 17 16:35:17.047295 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.047267    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe81b2e4-16f1-4338-b968-c9fcc694c207/kube-rbac-proxy-web/0.log"
Sep 17 16:35:17.086393 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.086297    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe81b2e4-16f1-4338-b968-c9fcc694c207/kube-rbac-proxy/0.log"
Sep 17 16:35:17.117292 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.117260    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe81b2e4-16f1-4338-b968-c9fcc694c207/kube-rbac-proxy-thanos/0.log"
Sep 17 16:35:17.147406 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.147383    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_fe81b2e4-16f1-4338-b968-c9fcc694c207/init-config-reloader/0.log"
Sep 17 16:35:17.190235 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.190208    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-rc7gk_694e4ba2-2ded-49b7-9084-0a86210086c4/prometheus-operator/0.log"
Sep 17 16:35:17.215532 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.215501    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-rc7gk_694e4ba2-2ded-49b7-9084-0a86210086c4/kube-rbac-proxy/0.log"
Sep 17 16:35:17.267009 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.266974    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-d5587_ee68ea93-3b67-41d4-a7af-5505da06106d/prometheus-operator-admission-webhook/0.log"
Sep 17 16:35:17.421060 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.420978    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5557b7d44b-2jfp6_d0dd1976-101f-4941-a5c2-2341bcddb7ea/thanos-query/0.log"
Sep 17 16:35:17.462504 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.462448    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5557b7d44b-2jfp6_d0dd1976-101f-4941-a5c2-2341bcddb7ea/kube-rbac-proxy-web/0.log"
Sep 17 16:35:17.499193 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.499144    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5557b7d44b-2jfp6_d0dd1976-101f-4941-a5c2-2341bcddb7ea/kube-rbac-proxy/0.log"
Sep 17 16:35:17.528322 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.528298    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5557b7d44b-2jfp6_d0dd1976-101f-4941-a5c2-2341bcddb7ea/prom-label-proxy/0.log"
Sep 17 16:35:17.559195 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.559168    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5557b7d44b-2jfp6_d0dd1976-101f-4941-a5c2-2341bcddb7ea/kube-rbac-proxy-rules/0.log"
Sep 17 16:35:17.592074 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:17.592052    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5557b7d44b-2jfp6_d0dd1976-101f-4941-a5c2-2341bcddb7ea/kube-rbac-proxy-metrics/0.log"
Sep 17 16:35:19.917514 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.917477    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"]
Sep 17 16:35:19.917994 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.917953    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" containerName="gather"
Sep 17 16:35:19.917994 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.917971    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" containerName="gather"
Sep 17 16:35:19.918121 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.917997    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" containerName="copy"
Sep 17 16:35:19.918121 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.918006    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" containerName="copy"
Sep 17 16:35:19.918121 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.918073    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" containerName="copy"
Sep 17 16:35:19.918121 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.918088    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6df12d7-8db1-418e-8c3e-2f3c5434f779" containerName="gather"
Sep 17 16:35:19.921159 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.921137    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:19.925770 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.925749    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bp66r\"/\"default-dockercfg-sz5vk\""
Sep 17 16:35:19.926206 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.926188    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bp66r\"/\"openshift-service-ca.crt\""
Sep 17 16:35:19.927794 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.927777    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bp66r\"/\"kube-root-ca.crt\""
Sep 17 16:35:19.943719 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.943692    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"]
Sep 17 16:35:19.999781 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.999746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-podres\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:19.999781 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.999785    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mgjc\" (UniqueName: \"kubernetes.io/projected/85fb8ee8-8477-446f-bb6e-a3b800a3868f-kube-api-access-4mgjc\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.000005 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.999813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-lib-modules\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.000005 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.999898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-sys\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.000005 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:19.999958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-proc\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.100922 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.100883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-proc\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.101100 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.100946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-podres\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.101100 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.100972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mgjc\" (UniqueName: \"kubernetes.io/projected/85fb8ee8-8477-446f-bb6e-a3b800a3868f-kube-api-access-4mgjc\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.101100 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.100993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-lib-modules\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.101100 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.101015    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-proc\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.101100 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.101082    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-sys\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.101402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.101109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-lib-modules\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.101402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.101117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-podres\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.101402 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.101167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85fb8ee8-8477-446f-bb6e-a3b800a3868f-sys\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.113996 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.113962    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mgjc\" (UniqueName: \"kubernetes.io/projected/85fb8ee8-8477-446f-bb6e-a3b800a3868f-kube-api-access-4mgjc\") pod \"perf-node-gather-daemonset-fnnwv\" (UID: \"85fb8ee8-8477-446f-bb6e-a3b800a3868f\") " pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.231324 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.231231    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:20.503942 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:20.503734    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"]
Sep 17 16:35:20.506615 ip-10-0-140-157 kubenswrapper[2581]: W0917 16:35:20.506590    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod85fb8ee8_8477_446f_bb6e_a3b800a3868f.slice/crio-6e32991f520070f8a27f8c3a616418905fe2a3b661633e4d7ef8504b6b569b36 WatchSource:0}: Error finding container 6e32991f520070f8a27f8c3a616418905fe2a3b661633e4d7ef8504b6b569b36: Status 404 returned error can't find the container with id 6e32991f520070f8a27f8c3a616418905fe2a3b661633e4d7ef8504b6b569b36
Sep 17 16:35:21.417562 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:21.417524    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv" event={"ID":"85fb8ee8-8477-446f-bb6e-a3b800a3868f","Type":"ContainerStarted","Data":"ce37b0589765255e664f5f8eb972611a9652a6fd3104cf40b145e70dfa46e013"}
Sep 17 16:35:21.417994 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:21.417573    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv" event={"ID":"85fb8ee8-8477-446f-bb6e-a3b800a3868f","Type":"ContainerStarted","Data":"6e32991f520070f8a27f8c3a616418905fe2a3b661633e4d7ef8504b6b569b36"}
Sep 17 16:35:21.417994 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:21.417616    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:21.435147 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:21.435090    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv" podStartSLOduration=2.435072114 podStartE2EDuration="2.435072114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 16:35:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 16:35:21.433313856 +0000 UTC m=+3412.356888434" watchObservedRunningTime="2026-09-17 16:35:21.435072114 +0000 UTC m=+3412.358646680"
Sep 17 16:35:21.592206 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:21.592173    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sfthf_0dc7d157-3364-4dde-93a0-06e97da1db07/dns/0.log"
Sep 17 16:35:21.617828 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:21.617798    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sfthf_0dc7d157-3364-4dde-93a0-06e97da1db07/kube-rbac-proxy/0.log"
Sep 17 16:35:21.643476 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:21.643446    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7gbxk_d033245e-d93b-4b38-b1c8-62351fd7cfc1/dns-node-resolver/0.log"
Sep 17 16:35:22.177333 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:22.177252    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-48hkv_6b5af873-2e95-452f-9eb6-936d70e56b0a/node-ca/0.log"
Sep 17 16:35:23.060584 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:23.060550    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-648db6667b-2h95h_b4db3ce1-346c-42de-8ca6-240881068330/router/0.log"
Sep 17 16:35:23.512102 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:23.512026    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jdr9w_d25033f9-7df1-47bb-a41a-3e3a57d36da5/serve-healthcheck-canary/0.log"
Sep 17 16:35:24.292106 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:24.292078    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wn4ng_3f4856dd-06b6-4154-bcdb-569255b6c63a/kube-rbac-proxy/0.log"
Sep 17 16:35:24.324149 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:24.324125    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wn4ng_3f4856dd-06b6-4154-bcdb-569255b6c63a/exporter/0.log"
Sep 17 16:35:24.361184 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:24.361159    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wn4ng_3f4856dd-06b6-4154-bcdb-569255b6c63a/extractor/0.log"
Sep 17 16:35:27.274773 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:27.274732    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vj6dr_eb4bc29a-b99e-4107-964a-916507799d0c/s3-init/0.log"
Sep 17 16:35:27.294875 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:27.294838    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-vj6dr_eb4bc29a-b99e-4107-964a-916507799d0c/download-hf-model/0.log"
Sep 17 16:35:27.333192 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:27.333157    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-rmkx2_f8611437-ad9d-4aa3-bb8d-6bab1452ac35/s3-tls-init-custom/0.log"
Sep 17 16:35:27.359571 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:27.359533    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-rmkx2_f8611437-ad9d-4aa3-bb8d-6bab1452ac35/download-hf-model/0.log"
Sep 17 16:35:27.399852 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:27.399823    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-vrjqm_672ade60-2979-4e61-aab5-ff3652445d8d/s3-tls-init-serving/0.log"
Sep 17 16:35:27.419125 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:27.419099    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-vrjqm_672ade60-2979-4e61-aab5-ff3652445d8d/download-hf-model/0.log"
Sep 17 16:35:27.430773 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:27.430748    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bp66r/perf-node-gather-daemonset-fnnwv"
Sep 17 16:35:32.173636 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:32.173605    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-2ksvk_f6874331-4a57-439f-9665-0a269c3f40f4/migrator/0.log"
Sep 17 16:35:32.202488 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:32.202442    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-2ksvk_f6874331-4a57-439f-9665-0a269c3f40f4/graceful-termination/0.log"
Sep 17 16:35:33.660092 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:33.660063    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8b6mj_015a1c41-25ba-4b7f-a281-626eeee00f40/kube-multus/0.log"
Sep 17 16:35:33.934142 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:33.934072    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dcjdl_e9f08147-9fea-4b2f-a2d7-59d0796228b2/kube-multus-additional-cni-plugins/0.log"
Sep 17 16:35:33.963553 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:33.963516    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dcjdl_e9f08147-9fea-4b2f-a2d7-59d0796228b2/egress-router-binary-copy/0.log"
Sep 17 16:35:33.995159 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:33.995127    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dcjdl_e9f08147-9fea-4b2f-a2d7-59d0796228b2/cni-plugins/0.log"
Sep 17 16:35:34.025860 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:34.025834    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dcjdl_e9f08147-9fea-4b2f-a2d7-59d0796228b2/bond-cni-plugin/0.log"
Sep 17 16:35:34.054845 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:34.054818    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dcjdl_e9f08147-9fea-4b2f-a2d7-59d0796228b2/routeoverride-cni/0.log"
Sep 17 16:35:34.081094 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:34.081075    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dcjdl_e9f08147-9fea-4b2f-a2d7-59d0796228b2/whereabouts-cni-bincopy/0.log"
Sep 17 16:35:34.113849 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:34.113820    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dcjdl_e9f08147-9fea-4b2f-a2d7-59d0796228b2/whereabouts-cni/0.log"
Sep 17 16:35:34.493571 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:34.493533    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lhp55_4421c85c-5367-4632-b2c3-f91661885873/network-metrics-daemon/0.log"
Sep 17 16:35:34.523068 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:34.523032    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lhp55_4421c85c-5367-4632-b2c3-f91661885873/kube-rbac-proxy/0.log"
Sep 17 16:35:35.427411 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:35.427380    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-486bt_44355a48-2b12-4c88-8567-c926f8ba49f5/ovn-controller/0.log"
Sep 17 16:35:35.445958 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:35.445922    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-486bt_44355a48-2b12-4c88-8567-c926f8ba49f5/ovn-acl-logging/0.log"
Sep 17 16:35:35.475198 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:35.475170    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-486bt_44355a48-2b12-4c88-8567-c926f8ba49f5/kube-rbac-proxy-node/0.log"
Sep 17 16:35:35.501508 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:35.501478    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-486bt_44355a48-2b12-4c88-8567-c926f8ba49f5/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 16:35:35.536301 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:35.536270    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-486bt_44355a48-2b12-4c88-8567-c926f8ba49f5/northd/0.log"
Sep 17 16:35:35.598094 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:35.598066    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-486bt_44355a48-2b12-4c88-8567-c926f8ba49f5/nbdb/0.log"
Sep 17 16:35:35.663631 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:35.663552    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-486bt_44355a48-2b12-4c88-8567-c926f8ba49f5/sbdb/0.log"
Sep 17 16:35:35.765783 ip-10-0-140-157 kubenswrapper[2581]: I0917 16:35:35.765749    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-486bt_44355a48-2b12-4c88-8567-c926f8ba49f5/ovnkube-controller/0.log"